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ABSTRACT 


Advances  in  weapon  systems  technology  creates  the  potential 
for  increased  warfighting  capability.  These  advances 
simultaneously  create  the  need  for  effective  simulation  systems 
of  these  contemporary  technologies.  The  credibility  and 
capability  of  these  weapons  systems  Models  and  Simulation  (M&S) 
are  evaluated  by  a  Verification  and  Validation  (V&V)  process, 
typically  performed  during  the  system  development  and  subsequent 
Developmental  Testing  (DT) .  The  tactical  effectiveness  and 
suitability  of  the  integrated  system  are  then  evaluated  through 
Operational  Testing  (OT) .  Historically,  the  tasks  associated 
with  DT  and  OT  are  performed  by  separate  organizations  in 
isolation. 

This  thesis  proposes  a  methodology  for  the  Verification  and 
Validation  of  the  weapons  systems  models  implicit  in  the  AH-64D 
Longbow  Apache  Tactical  Engagement  Simulation  (TES)  System.  In 
addition,  this  thesis  develops  a  V&V  plan  to  evaluate  the 
simulation  provided  by  the  integrated  Longbow  TES  system.  The 
design  of  this  plan  provides  for  the  simultaneous  collection  of 
OT  data  to  support  system  suitability  evaluation.  This  will 
reduce  future  OT  requirements,  thus  decreasing  the  time  required 
for  the  acquisition  cycle.  This  proposition  of  performing  the  . 
TES  V&V  as  a  combination  of  DT  (V&V)  and  OT  supports  the  rapid 
prototyping  philosophy  which  is  useful  in  proving  the  concepts  of 
new  technology  and  complex  systems . 
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PART  ONE 


Part  One  explains  the  evolution  of  the  need  for  a 
Tactical  Engagement  Simulation  (TES)  System  for  the  AH-64D 
Longbow  Apache  Attack  Helicopter,  as  well  as  its  functions, 
elementary  hardware,  and  software. 
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CHAPTER  1 


INTRODUCTION 


OVERVIEW 

Advances  in  weapon  systems  technology  creates  the  potential 
for  increased  warfighting  capability.  These  advances 
simultaneously  create  the  need  for  effective  simulation  systems 
of  these  contemporary  technologies.  Meeting  the  simulation 
requirements  ensures  that  military  training  and  warfighting 
strategies  are  in  alignment,  thus  promoting  the  U.  S.  Army's 
"train-as-you-fight"  objective.  The  evaluation  of  Tactical 
Engagement  Simulation  (TES)  systems  in  a  realistic  operational 
environment  is  a  formidable  task.  As  part  of  Developmental 
Testing  (DT) ,  a  weapon  system  simulation  device  and  its  implicit 
weapon  system  models  must  be  evaluated  in  terms  of  the  design 
intentions;  further  testing  is  then  required  to  determine  their 
performance  in  an  operational  environment.  The  credibility  and 
capability  of  these  models  are  typically  assessed  by  a 
Verification  and  Validation  (V&V)  process.  The  V&V  is  normally 
performed  during  the  system  development  and  subsequent 
Developmental  Testing.  The  tactical  effectiveness  and 
suitability  of  the  integrated  system  is  then  evaluated  through 
Operational  Testing  (OT) .  Developmental  Tests  and  Operational 
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Tests  encompass  different  tasks;  historically,  the  tasks 
associated  with  DT  and  OT  are  performed  in  isolation. 

Considering  the  present  day  fiscal  constraints  placed  on  defense 
acquisition,  it  makes  sense  to  accomplish  the  requirements  of  the 
V&V  in  conjunction  with  OT  if  the  situation  allows. 


PURPOSE 

This  thesis  proposes  a  methodology  for  the  Verification  and 
Validation  of  the  weapons  systems  models  implicit  in  the  AH-64D 
Longbow  Apache  Tactical  Engagement  Simulation  (TES)  System.  In 
addition,  this  thesis  develops  a  V&V  plan  to  evaluate  the 
simulation  provided  by  the  proposed  integrated  Longbow  TES 
system.  The  design  of  this  plan  provides  for  the  simultaneous 
collection  of  OT  data  used  to  support  the  evaluation  of  system 
suitability  to  reduce  the  future  OT  requirements,  thus  decreasing 
the  time  required  for  the  acquisition  cycle  and  conserving 
resources.  The  resulting  test  plan,  presented  in  Appendix  B., 
consists  of  a  description  of  the  required  tests,  the  conditions 
under  which  the  system  is  to  be  tested,  a  statement  of  the  test 
criteria,  and  a  data  management  scheme.  V&V  activities  can  span 
a  broad  spectrum  of  issues.  Consequently,  the  Verification  and 
Validation  techniques  are  specific  to  the  Longbow  TES  System  and 
may  be  altered  for  similar  simulation  applications.  This 
approach  can  be  expanded  in  detail  and  applied  to  future  V&V 
efforts  if  required. 
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ORGANIZATION 


This  thesis  is  divided  into  three  parts.  Part  One 
describes  the  development,  functions,  and  the  purpose  of  the 
Longbow  Tactical  Engagement  Simulation  System. 

Part  Two  provides  the  description  of  the  requirements  for 
Verification  and  Validation,  and  proposed  methodologies  for  V&V 
of  the  Longbow  TES . 

Part  Three  presents  the  development  of  a  fundamental  test 
plan  by  which  a  team  can  qualitatively  and  quantitatively 
evaluate  the  Longbow  TES  models  and  the  integrated  system  in  an 
operational  environment.  The  methodology  is  founded  upon  the 
Army' s  approach  to  conducting  materiel  systems  evaluations  and  is 
aimed  at  inspiring  thought  on  how  to  accurately  assess  the 
Longbow  TES  models  when  faced  with  a  prescribed  test  schedule  and 
environment. 
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CHAPTER  2 


OVERVIEW  OF  ARMY  TACTICAL  ENGAGEMENT  SIMULATION 


TRAINING  PHILOSOPHY 

The  U.  S.  Army  trains  by  using  realistic,  live  maneuver 
exercises  as  a  primary  means  of  meeting  combat  readiness 
standards.  These  exercises  involve  combined  arms  training  at  the 
Army's  Combat  Training  Centers  (CTCs)  and  at  the  individual 
units'  home  stations.  Soldiers  train  using  real  and  surrogate 
weapons  systems  operating  in  authentic  combat  conditions. 
Competing  demands  for  shrinking  resources  during  the  post  Cold 
War  era  have  dictated  the  need  to  meet  readiness  objectives  at  a 
lower  cost.  As  a  result,  the  Army  of  the  late  1990s  was  faced 
with  reducing  its  reliance  on  traditional  resource-intensive 
operations  and  implementing  new  training  methodologies .  The 
resulting  methodology  combined  field  training  exercises  with 
virtual  and  constructive  battle  simulations  to  effectively  train 
to  standards  at  a  lower  resource  cost.  However,  synthetic 
environments  lack  the  realism  generated  by  maneuvering  forces 
under  representative  battlefield  conditions.  The  virtual  and 
constructive  simulations  can  at  best  augment  live  maneuver 
training  while  never,  fully  replacing  it  as  the  foundation  of  Army 
training.  Live  exercises,  executed  in  field  conditions  using 
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tactical  equipment,  are  enhanced  by  simulation  and  simulators. 

One  example  is  Tactical  Engagement  Simulation  (TES)  which 
replicates  weapons  actions  and  effects  under  combat  conditions. 
The  Army's  TES  training  methodology  is  characterized  by  "...the 
free  interplay  of  two  forces  using  a  Real  Time  Casualty 
Assessment  (RTCA)  system  that  reinforces  training  tasks  through 
immediate  feedback  response  to  correct  individual  and  collective 
task  accomplishment"  (TRADOC,  1998b,  1-3) . 

CURRENT  AVIATION  TES  SYSTEMS 

The  current  baseline  of  Tactical  Engagement  Simulation  used 
for  air  to  ground  Force-on-Force  ( FOF)  exercises  at  the  CTCs 
provides  for  instrumented  terrain  and  the  use  of  the  Multiple 
Integrated  Laser  Engagement  System/Air  to  Ground  Engagement 
System  II  (MILES/AGES  II)  and  the  Simulated  Area  Weapons  Effects 
(SAWE)  system.  The  MILES/AGES  II  simulates  in  real  time, 
tactical  engagements  of  select  rotary  wing  aircraft  and  direct- 
fire  ground  weapon  systems  by  the  use  of  Line-of-Sight  (LOS) 
laser  transmitters  and  receivers  to  pair  shooters  and  targeted 
vehicles.  Casualty  and  damage  assessment  for  the  aircraft  and 
ground  vehicles  are  based  on  their  vulnerabilities  to  various 
direct  fire  weapons;  the  MILES  detectors  recognize  the  type  of  • 
weapon  represented  by  the  laser  signal  fired  and  apply  the 
appropriate  probability  of  kill  (Pk)  for  an  engagement.  Audio 
and  visual  cues  to  the  operators  of  the  equipment  involved  in  the 
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engagement  identify  results  as  near-miss,  hit,  or  kill.  The 
MILES  devices  can  interoperate  with  other  simulators  and  external 
instrumentation  systems  to  collect  training  performance  data. 
These  data  serve  as  a  primary  source  of  information  for  After 
Action  Reviews  (AARs)  for  maneuver  CTC  exercises.  MILES/AGES  II 
is  currently  fielded  both  to  units  and  to  the  maneuver  CTCs  for 
limited  types  of  aircraft.  The  SAWE  uses  Radio  Frequency  (RF) 
and  Global  Positioning  System  (GPS)  technology  to  simulate  the 
effects  of  direct  and  indirect  fire  weapons  and  is  currently 
fielded  at  the  maneuver  CTCs. 

THE  NEED  FOR  LONGBOW  TES 

The  Longbow  was  scheduled  to  participate  in  the  Army's 
Advanced  Warfighting  Experiment  (AWE)  "Task  Force  XXI"  in  1997; 
however,  an  acceptable  amount  of  training  fidelity  could  not  be 
realized  since  the  aircraft's  non  Line-of-Sight  missile 
capability  would  have  to  be  excluded  from  the  battle.  Existing 
Real  Time  Casualty  Assessment  TES  systems  at  all  of  the  Army's 
three  major  maneuver  Combat  Training  Centers  and  home  stations 
are  currently  unable  to  support  the  full  capabilities  of  the 
Longbow's  weapons  systems.  Both  the  MILES  and  SAWE  do  not  have 
growth  provisions  for  emerging  non-Line-of-Sight  technology.  The 
MILES  II,  is  scheduled  to  be  replaced  within  the  next  ten  years 
by  an  upgraded  version  called  MILES  2000.  This  new  version  will 
also  be  limited  to  use  with  LOS  weapons  systems.  Reduced  program 
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funding  for  SAWE  prohibits  the  integration  of  additional  vehicle 
kits,  weapons,  or  munitions.  The  growth  of  new  weapons  systems 
technology  and  capabilities  are  increasing  faster  than  the 
advances  in  the  training  arena.  •  Consequently,  the  present 
generation  of  Tactical  Engagement  Simulation  systems  are 
diminishing  as  effective  training  tools. 

FUTURE  TES 

"Force  XXI",  a  term  used  to  define  the  modern,  streamlined 
Army  of  the  21st  century,  is  characterized  by  the  use  of 
predominantly  digital  information  systems  technology  throughout 
the  battlefield.  Modernization  initiatives  projected  for  Force 
XXI  have  currently  identified  275  systems  or  munitions  which  have 
potential  use  in  live-fire  exercises.  The  goal  of  generating 
realistic  simulated  engagement  training  for  the  Longbow  weapons 
systems  at  the  training  centers  and  home  stations  poses  an 
additional  challenge  since  each  CTC  location  differs  in  terms  of 
instrumentation,  weapon  system  simulation  capability,  data 
message  formats,  terrain,  size,  battlegroup  makeup,  and  mission. 
As  a  solution  to  the  weapon  system/training  environment 
disparity,  the  Army  proposed  an  operational  concept  for  the 
integration  and  standardization  of  Tactical  Engagement  Simulation 
throughout  the  Army.  This  concept  is  outlined  in  the  TES  Master 
Plan  (MP) .  The  Master  Plan  earmarks  the  Longbow  Apache  TES 
System  to  be  fielded  at  the  Joint  Readiness  Training  Center 
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( JRTC) ,  the  Combat  Maneuver  Training  Center  (CMTC) ,  the  National 
Training  Center  (NTC) ,  and  at  select  unit  home  stations.  The  TES 
Master  Plan  defines  numerous  research  efforts  which  assess  and 
project  the  impacts  of  future  technologies  and  Army  force 
modernization  initiatives  on  live  training  support.  Two  proposed 
research  initiatives.  The  Advanced  Tactical  Engagement 
Simulations  Concepts  (ATESC)  and  Future  TES,  will  define  the 
feedback  needed  to  train  Force  XXI  and  will  identify  the  detailed 
requirements  for  future  TES  development  and  use.  These 
initiatives  however  are  programmed  for  Fiscal  Year  (FY) 99-00  and 
FY01-02  research  respectively.  Therefore,  in  the  interim, 
materiel,  training,  and  combat  developers  must  work  closely  on 
the  development  efforts  for  a  new  generation  of  TES  to  ensure  the 
development  is  aligned  with  the  Combined  Arms  Training  Strategy 
(CATS)  and  compatible  with  the  Army's  anticipated  training 
infrastructures.  The  training  and  combat  developers  must  define 
the  requirements  of  new  training  systems  based  upon  the  general 
guidelines  set  forth  in  the  TES  Master  Plan.  In  order  to  direct 
a  successful  materiel  development,  the  workgroups  formed  between 
the  developers  should  be  intimately  involved  with  the  process 
from  its  inception  throughout  the  entire  acquisition  lifecycle. 


LONGBOW  TES  DEVELOPMENT 

Inter-Coastal  Electronics  (ICE) ,  Incorporated  of  Mesa, 
Arizona  was  awarded  contracts  through  the  Army's  Communications 
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and  Electronics  Command  (CECOM)  and  the  Aviation  and  Missile 
Command  (AMCOM)  to  conduct  a  proof  of  concept  for  a  unit 
Homestation  Training  Instrumentation  (HTI)  system.  The 
Homestation  Training  Instrumentation  initiative  aims  to  design  a 
number  of  standard,  fixed  sight,  state-of-the-art  training 
centers  capable  of  supporting  new  weapons  capabilities  while 
providing  for  a  growth  potential.  The  HTI  will  be  an  integrated 
system  of  computer  software  and  hardware,  workstations, 
databases,  voice  and  video  recorders,  production  and  presentation 
equipment,  interface  devices,  and  communication  systems.  The 
system  is  to  be  capable  of  automated  data  collection  and  analysis 
to  control  tactical  exercises  and  provide  training  performance 
feedback.  The  initial  demonstration  of  this  system  was  called 
Collective  Helicopter  Operations  in  a  Combat  Environment 
(CHOICE) .  The  functions,  interfaces,  and  equipment  for  a 
proposed  Tactical  Engagement  Simulation  system  for  t'he  Longbow 
Apache  were  conceived  and  developed  under  the  CHOICE  system  Phase 
I  demonstration  during  the  first  quarter  of  FY98.  ICE  and  the 
Boeing  Helicopter  Company  are  continuing  the  development  of  the 
Longbow  TES  System  as  an  extension  of  the  CHOICE  capabilities. 

VERIFICATION  AND  VALIDATION  EFFORTS 

As  the  proponent  for  the  Longbow  TES  system.  The  U.  S.  Army 
Aviation  Center  (USAAVNC) ,  U.  S.  Army  Training  and  Doctrine 
Command  (TRADOC)  is  examining  the  impact  of  new  materiel  systems 
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development  on  combat  aviation  training.  As  a  result,  the 
Program  Executive  Office  (PEO) ,  Longbow  Apache,  directed  an 
independent  Verification  and  Validation  (V&V)  for  the  proposed 
TES  system.  The  Program  Manager  (PM)  requested  assistance  from 
Quality  Research  (QR) ,  Incorporated  and  the  University  of 
Tennessee  Space  Institute  (UTSI)  flight  systems  research  group  to 
examine  the  proposed  TES  system  at  its  current  stage  of 
development.  That  support  was  provided  under  contract  number 
F40600-94-D-0001 .  The  V&V  effort  for  the  integrated  Longbow  TES 
System  was  to  be  initiated  during  the  August  1998  timeframe  at 
Fort  Hood,  Texas,  with  it  installed  on  Lot  1  aircraft;  Serial  No: 
96-5001  through  96-5027,  97-5025,  and  97-5027.  The  V&V  plan  was 
conceived,  but  due  to  scheduling  the  fully  installed  TES  system 
was  not  available  for  operation.  The  V&V  effort  is  projected  to 
continue  during  FY-00  on  Lot  4  aircraft. 
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CHAPTER  3 


SYSTEMS  DESCRIPTIONS 

AH-64D  LONGBOW  APACHE  ATTACK  HELICOPTER 

The  AH-64D  Longbow  Apache  helicopter  is  a  twin  engine, 
tandem  seat  aerial  weapons  platform  (DA,  1988,  2-1) .  Wings 
attached  to  the  center  fuselage  accommodate  a  total  of  four 
external  stores /weapons  pylons  with  hydraulic  and  electrical 
quick-disconnects .  The  Longbow  is  designed  for  multiple  combat 
missions  to  include  attack,  reconnaissance,  and  security.  The 
current  AH-64D  weapons  subsystems  include  the  M139  Area  Weapon 
System  (AWS)  which  includes  the  M230E1  30mm  gun.  Longbow  Hellfire 
Modular  Missile  System  (LBHMMS)  with  Radio  Frequency  (RF)  and 
Semi -Active  Laser  (SAL)  guided  missiles,  and  the  M140  Aerial 
Rocket  System  (ARS) .  A  list  of  standard  weapons  configurations 
and  associated  mission  profiles  is  presented  as  Table  3-1.  A 
more  detailed  description  of  the  aircraft  weapons  systems  are 
included  in  Appendix  A. 

The  Longbow  is  an  improved  variation  of  the  AH-64A  Apache. 
The  significant  technological  improvement  is  the  incorporation  of 
a  mast-mounted,  millimeter  wave  Fire  Control  Radar  (FCR) .  The 
FCR  is  an  air/ground,  targeting  system  used  in  conjunction  with 
the  Hellfire  Missile  system  to  allow  the  aircrew  to  employ  the 
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Table  3-1 


AH-64  Mission  Configurations 


(Adapted  from:  AttacK  Helicopter  Operations,  p.  A-5) 


Pylons 

Load 

Left 

Outboard 

Left 

Inboard 

Right 

Inboard 

Right 

Outboard 

Role 

A 

4  Hellfire 

19  Rockets 

19  Rockets 

4  Hellfire 

Recon/Attack 

B 

4  Hellfire 

fuel  tank 
(230)  gal. 

19  Rockets 

4  Hellfire 

Recon/Attack 

C 

4  Hellfire 

4  Hellfire 

4  Hellfire 

4  Hellfire 

Attack 

D 

4  Hellfire 

4  Hellfire 

fuel  tank 
(230)  gal. 

4  Hellfire 

Attack 

E 

19  Rockets 

19  Rockets 

19  Rockets 

19  Rockets 

Recon/Attack, 

Security 

F 

19  Rockets 

fuel  tank 
(230)  gal. 

19  Rockets 

19  Rockets 

Recon/Attack 

G 

4  Hellfire 

19  Rackets 

fuel  tank 
(230)  gal. 

4  Hellfire 

Recon 

Note : 

All  loads  include  1200  rounds  of  30mm  ammunition 

AGM-114K  RF  Hellfire  Missile  against  non-Line-of-Sight  targets. 

It  is  capable  of  operation  during  day  or  night  and  in  adverse 
weather  conditions.  Other  improvements  include  additional  power, 
expanded  avionics  bays,  upgraded  systems  processors,  integrated 
avionics,  refined  crew  stations,  and  an  Improved  Data  Modem  (IDM) 
that  allows  secure  target  and  situation  data  transfer.  The  U.  S. 
Army  accepted  the  first  production  model  AH-64D  at  the  Boeing 
Company  (formerly  McDonnell  Douglas)  facility  in  Mesa,  Arizona, 
on  March  21,  1997.  The  Longbow  Apache  is  depicted  in  Figure  3-1. 
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Figure  3-1 


AH-64D  Longbow  Apache 

(Source:  Boeing,  1998) 

TACTICAL  ENGAGEMENT  SIMULATION  (TES)  SYSTEM  DESCRIPTION 
General 

The  Longbow  TES  System  is  designed  to  perform  Tactical 
Engagement  Simulation  using  Global  Positioning  System  geometric 
pairing  technology  and  be  interoperable  with  the  current  Multiple 
Integrated  Laser  Engagement  System/Air-to-Ground  Engagement 
Simulation  (MILES/AGES  II) .  The  TES  system  is  designed  to 
faithfully  replicate  the  capabilities  of  all  on-board  weapons 


systems,  sensors,  and  designators  for  Force-on-Force  training. 

The  TES  System  is  designed  to  provide  for  the  following  weapon 
engagement  simulations:  GPS  geometric  pairing  of  the  Radio 
Frequency  Hellfire  missile,  Semi-Active  Laser  Hellfire  missile, 
Aerial  Rocket  System,  and  30mm  cannon.  It  is  to  also  include 
laser  simulation  of  the  SAL  Hellfire  missile  and  30mm  cannon 
using  eye-safe  weapon  system  laser  emitters.  The  TES  system  is 
projected  to  interface  with  the  instrumentation  systems  planned 
for  the  National  Training  Center,  Joint  Readiness  Training 
Center,  Combat  Maneuver  Training  Center,  and  select  unit  home 
station  locations.  Additionally,  the  TES  System  is  designed  to 
reduce  Longbow  Apache  Contractor  Logistic  Support  in  the  training 
environment  (MDHC,  1997,  1)  . 

TES  System  Instrumentation  Package 

The  TES  System  Instrumentation  Package  consists  of  the 
following  sub-elements:  ten  (10)  sets  of  the  Longbow  Apache  TES 
System  Aircraft  Instrumentation  Package  (AIP)  equipment,  one  (1) 
Central  Collection  Facility  (CCF)  van,  thirty-five  (35)  Target 
Instrumentation  Kits  to  be  used  as  a  credible  Opposing  Force 
(OPFOR) ,  four  (4)  Repeater  Units,  and  two  (2)  Training  Data 
Playback  Units  (TDPU) .  The  capabilities  from  Phase  I  CHOICE  will 
be  expanded  to  include  Real  Time  Casualty  Assessment.  Portions 
of  the  integrated  TES  System  Instrumentation  Package  will  be 
designed,  built,  tested,  and  fielded  initially,  while  other 
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portions  are  planned  as  optional  efforts  that  will  be  exercised 
later. 

Aircraft  Instrumentation  Package  (AIP) 

The  Longbow  TES  Aircraft  Instrumentation  Package  employs 
the  Army's  "embedded  training"  concept.  This  concept  requires 
that  a  major  portion  of  the  essential  training  functions  are 
built  into  the  aircraft  subsystems .  The  remainder  of  equipment 
and  software  are  installed  or  attached  to  the  aircraft  when 
needed,  and  removed  when  not  needed.  The  TES  AIP  consists  of  two 
separate  subsystems,  designated  as  the  "A-Kit"  and  "B-Kit". 

A-Kit  Description  The  A-kit  is  limited  to  modifications  that  are 
made  to  on-board,  organic  aircraft  software  in  order  to  conduct 
initialization  and  operation  of  TES,  and  Built  In  Test  (BIT) 
functions.  The  A-Kit  consists  of  software  modifications  to  the 
following  aircraft  Line  Replacement  Units  (LRUs)  that  support  TES 
implementation: 

•  Weapons  Processor  (WP) 

.  System  Processor  (SP) 

•  Display  Processor  (DP) 

.  Communications  Interface  Unit  (CIU) 

•  Radio  Frequency  Interferometer  (RFI) 

•  Radar  Warning  Receiver  (RWR) 
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TADS  Electronic  Unit  (TEU) 


B  Kit  Description  The  B  kit  is  comprised  of  the  following 
appended  hardware  components  plus  embedded  software: 

.  Smart  On-Board  Data  Interface  Module  (SMODIM) 

.  Training  Laser  Interface  Adapter  (TLIA) 

.  Data  Communications  Interface  (DCI) 

.  Indicator  Control  Unit  (ICU) 

TES  System  Training  Missile  (TTM) 

.  Laser  Rangefinder/Designator  (LRFD) 

.  Area  Weapon  System  Simulator  (30mm  Gun  Laser) 

The  integrated  TES  System  architecture  is  depicted  in  Figure  3-2; 
A  Kit  components  are  designated  by  an  "A"  and  B  Kit  components 
are  designated  by  a  "B".  A  detailed  description  and  photographs 
of  the  proposed  Longbow  TES  System  components  are  located  in 
Appendix  A. 
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Figure  3-2 

Longbow  TES  System  Architecture 

(Source:  Production  Longbow  Test  Plan,  Phase  II  of  TES  System, 

P-  3) 
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CHAPTER  4 


TACTICAL  ENGAGEMENT  SIMULATION  (TES)  AND  MODEL 


OPERATIONAL  CONCEPT 
Geometric  Pairing 

The  Longbow's  capability  to  employ  the  AGM-114K  RF  Hellfire 
Missile  against  non  Line-of-Sight  targets  renders  the  MILES/AGES 
II  incapable  of  performing  RTCA  commensurate  with  the  aircraft's 
capabilities.  Consequently,  the  Longbow  TES  System  will  employ  a 
concept  known  as  GPS  geometric  pairing  to  match  all  LOS  and  non- 
LOS  weapons  engagements.  Using  this  concept,  the  aircraft  and 
targeted  vehicles  are  paired  by  their  known  positions  as 
determined  by  GPS,  for  the  purpose  of  performing  RTCA.  The  AH- 
64D  is  equipped  with  a  dual  Embedded  GPS/INS  (EGI)  that  provides 
the  aircraft  position  and  velocity  information.  Raw  data  from 
the  EGI  in  the  helicopter  are  downlinked  to  a  ground  reference 
station,  as  depicted  in  Figure  4-1.  The  reference  station 
computes  the  GPS  pseudorange  corrections,  the  rate  of  change  of 
the  corrections,  and  then  applies  the  corrections  to  the 
aircraft's  measurements  to  compute  the  actual  time/space/position 
information.  This  method  offers  the  option  to  send  all  GPS 
geometric  pairing  RTCA  data  to  the  CTC  host  computer  where  the 
data  can  be  combined  with  other  range  data.  During  a  non-LOS  RF, 
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Hellfire  missile  engagement,  the  Longbow's  aimpoint  location  is 
determined  by  the  FCR  target  position  relative  to  the  aircraft 
position  provided  by  the  EGI .  "During  a  LOS  engagement,  the 
aimpoint  {range,  azimuth,  and  look-down  angle)  is  determined  by 
triangulation  referenced  to  the  aircraft  heading,  attitude,  and 
GPS  position.  The  aimpoint  information  is  extracted  from  the 
aircraft  bus  by  the  SMODIM  and  transmitted  by  the  Data 
Communications  Interface  (DCI)  to  the  range  data  management 
facility's  host  computer.  The  data  facility  then  places  the 
specific  weapons  footprint  over  the  aimpoint  and  calculates  the 
time  (UTC  time  stamped)  of  the  engagement  event.  The  data 


GPS  Geometric  Pairing  (Non-Line-of- Sight  Engagement) 
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facility  searches  back  in  time  history  and  queries  the  data  to 
determine  which  target  was  in  the  footprint  when  the  weapons 
trigger  was  pulled.  .  If  a  target  was  determined  to  be  in  the 
footprint  at  that  time,  the  targeted  vehicle  will  perform  the 
RTCA  computations  and  send  the  results  of  the  engagement  to  the 
data  management  facility  via  a  repeater  for  scoring.  Software 
upgrades  will  be  necessary  for  the  communication  networks  at  the 
CTCs  to  accept  the  Longbow  TES  data  message  structures.  While 
operating  at  a  unit's  homestation,  the  TES  System  is  designed  to 
allow  player  aircraft  to  act  as  repeaters  and  establish  its  own 
communication  network  by  use  of  the  CHOICE  System  telemetry  radio 
and  the  TES  System  Central  Collection  Facility.  The  Longbow  TES 
System  is  projected  to  operate  in  the  commercial  (902-926)  MHz 
band  at  a  data  rate  of  (115.2  KBaud) . 

WEAPONS  SIMULATION 
General 

The  TES  System  is  to  allow  all  weapons  to  be  emulated  when 
the  system  is  placed  in  the  "ready  mode".  The  emulation  will 
drive  the  display  symbology  to  replicate  actual  weapons  displays. 

Gun  Simulation 

The  gun  rounds  inventory  is  loaded  through  the  weapons  page 
by  the  flight  crew.  The  Weapons  Processor  (WP)  and  the  SMODIM 
decrement  the  rounds  fired  based  upon  the  burst  limit  and  trigger 
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pull  duration.  Upon  trigger  pull,  the  WP  commands  the  Training 
Laser  Interface  Adapter  (TLIA)  to  fire  the  gun  Flash  Weapons 
Effect  Signature  Simulation  (FlASHWESS) /LASER  mounted  on  the 
30mm  gun.  The  TLIA  provides  a  "GUN  LASER  PRESENT"  message  to  the 
WP  when  the  laser  is  installed  on  the  gun.  When  the  gun  is 
enabled,  the  WP  positions  the  gun  to  the  Line-of-Sight  (LOS)  of 
the  crewmember  that  has  selected  the  gun.  The  WP  applies 
fuselage  body  bending  corrections,  gun  boresight  corrections,  and 
parallax  corrections  when  determining  the  LOS.  The  display  in 
the  Display  Processor  (DP)  will  be  identical  to  that  of  an  actual 
gun  engagement.  The  System  Processor  will  generate  gun  sound 
effects  based  upon  WP  firing  data. 

Rocket  Simulation 

The  rocket  type,  quantity,  and  zone  are  automatically 
determined  by  the  SMODIM.  During  the  initialization  process,  the 
SMODIM  reads  the  inventory  previously  loaded  into  the  rocket  Load 
Maintenance  Panel  (LMP)  by  the  groundcrew.  A  trigger  pull 
results  in  the  rocket  inventory  being  decremented  based  upon  the 
zone,  type  fuze  selected,  and  trigger  pull  duration.  When  the 
rocket  system  is  enabled,  the  Weapons  Processor  positions  the 
rocket  steering  cursor  to  the  LOS  of  the  crewmember  who  actioned 
the  weapon;  the  aimpoint  is  computed  based  upon  the  selected  LOS. 
The  FLASHWESS  will  simulate  rocket  fire  effects  and  the  System 
Processor  (SP)  will  simulate  rocket  sound  effects.  Additionally, 
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the  pylons  will  articulate  as  they  would  with  the  actual  weapon 
system. 

Hellfire  Simulation 

The  missile  type  and  quantity  information  is  loaded  during 
the  inventory  initialization;  each  missile  type  is  emulated. 
Display  information  is  sent  to  the  DP  to  replicate  tactical 
displays.  Upon  trigger  pull,  the  Weapons  Processor  sends  the 
missile  launch  status,  tracking,  and  missile  time  of  flight  to 
the  DP  and  subsequently  decrements  the  missile  count. 

TES  MODEL 

A  preliminary  model  for  a  rocket  engagement  is  presented  as 
described  in  a  developmental  information  paper  prepared  by 
McDonnell  Douglas  Helicopter  Systems.  The  model  has  been 
translated  from  computer  code  and  is  presented  in  a  form  which 
allows  the  reader  to  understand  its  composition  and  the  relevance 
of  specific  event  data  required  for  a  simulated  weapons 
engagement  of  this  type.  One  example  illustrates  the  simulation 
sufficiently  to  provoke  thought  on  how  to  formulate  a  test  for 
model  fidelity. 

Geometrically  Paired  RTCA  Rocket  Engagement 

Cockpit  Displays  The  rockets  inventory,  quantity,  fuze  and 

warhead  type,  and  system  status  are  provided  to  the  aircrew  by 
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the  TES  System  through  the  rocket  system  controls  display  (Figure 
4-2)  . 

Preconditions  for  the  event  The  firing  restrictions  and 
limitations  imposed  by  the  aircraft  are  described  as  a  prelude  to 
an  example  rocket  engagement.  The  aircraft  will  restrict  the 
crew  from  firing  when  the: 

•  Aerial  Rocket  System  (ARS)  detects  a  failure 
.  Salvo  limit  is  reached 

•  Rocket  inventory  <  0 


Figure  4-2 

Aerial  Rocket  System  Controls 

(Source:  Operator's  Manual  for  AH-64D  Helicopter,  P.  '4-81) 


24 


The  target  zone  or  "target  footprint"  is  determined  by  the  rocket 
type  which  has  been  inventoried.  For  the  MK66  rocket  with  a 
Point  Detonating  (PD)  warhead,  ~the  target  zone  range  delta  (ART)  = 
125m,  and  the  target  zone  azimuth  delta  (Aazt)  =  23mR.  This  means 
the  TES  System  constructs  a  virtual  box  that  is  125  meters  deep 
and  23  milliradians  wide  that  defines  the  impact  zone  of  the 
rockets.  For  the  MK66  rocket  with  the  Multi-Purpose  submunitions 
(MP)  warhead,  the  (ART)  =  200m,  and  (Aazt)  =  34mR.  Table  4-1 
depicts  the  probability-of-hit  (Ph)  as  a  function  of  the  target 
characteristics  ("hard-skin"  vs.  "soft-skin"),  and  warhead  type. 
For  actual  aerial  rocket  shots,  angular  errors  produce  a  larger 
and  larger  footprint  with  increased  range,  thus  the  widening  of 
the  target  zone  with  increased  range  is  a  good  modeling 
assumption.  Referencing  table  4-1,  the  Ph is  independent  of  the 
range  when  the  constraints  have  been  met  with  no  limitations .  It 
is  however,  contingent  upon  the  number  of  rocket  pairs  fired 
during  an  engagement  sequence.  For  each  additional  pair  of 
rockets  launched  during  an  engagement,  the  probability-of-hit  is 
defined  by  the  following  binomial  expansion  in  disguise: 

P.  —  1  —  / 1  —  D  \  n  pairs 

n  pairs  -1-  \  1  pair / 

This  expression  implies  that  regardless  of  how  many  pairs  are 
fired,  the  Ph  reflects  the  probability  of  at  least  one  pair 
hitting  the  target.  The  expression  says  nothing  about  the 
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Table  4-1 


Rocket  Event  Probability  of  Hit  (Ph)  for  Pair  Fired 

(Adapted  from:  TES  System  Geometric  Real  Time  Casualty 
Assessment  (RTCA) ,  p.  23) 


Rocket  Type ,  6PD 

Rocket  Type ,  6MP 

Target  Zone  Azimuth 
Delta  (Ari)  ,  mR 

23mR 

34mR 

Target  Zone  Range 

Delta  (Aaz?)  ,  mR 

125m 

200m 

Range,  m 

(Ph) 

soft 

(Ph) 

hard 

Target 

A  <m) 

(Ph) 

soft 

(Ph) 

hard 

500  -  1500 

0.035 

0.035 

540 

0.15 

0 

1500  -  2000 

0.035 

0.035 

630 

0.15 

0 

2500  -  3500 

0.035 

0.035 

520 

0.15 

0 

3500  -  4500 

0.035 

0.035 

330 

0.15 

0 

4500  -  5500 

0.035 

0.035 

240 

0.15 

0 

5500  -  6500 

0.035 

0.035 

160 

0.15 

0 

6500  -  7500 

0.035 

0.035 

105 

0.15 

0 

>7500 

0.035 

0.035 

70 

0.15 

0 

probability  of  hitting  with  2,3  ...  or  n  pairs,  or  even  killing 
the  target.  Understanding  this  fundamental  Ph  characteristic 
will  be  important  for  constructing  the  test  to  verify  the  model. 

The  model  also  imposes  a  degradation  in  Ph  when  the 
commanded  pylons  position  is  at  the  limit.  When  the  crew 
receives  an  "AT  LIMITS"  message,  the  target  range  is  re¬ 
calculated  by  subtracting  a  Range  Delta  from  the  crew's  selected 
range.  The  Range  Delta  is  defined  as : 

Range  Delta  =  (Pylon  Elevation  -  4°)  x  Target  A 

where:  Target  A  =  (value  from  Table  4-1) 
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The  Engagement  Sequence  When  the  rockets  are  selected  and  the 
weapons  trigger  is  pulled  and  the  aircraft  detects  no  firing 
restrictions,  then  the  weapon  event  is  initiated.  When  the 
firing  terminates  and  the  rocket  type  has  been  detected  as  6PD  or 
6MP,  then  the  range  delta  is  applied  if  necessary.  The  TADS 
Electronic  Control  Unit  converts  the  MUX  data  to  a  weapon  event 
message  and  transmits  the  message  to  all  players.  The  players 
determine  the  Longbow  Apache  rocket  weapon  event  from  the  data. 

If  a  player  was  in  the  target  zone  (referenced  to  the  aircraft 
GPS  position)  at  the  weapon  event  time,  then  the  player  continues 
to  process  the  message.  The  player  or  players  in  the  target  zone 
perform (s)  the  RTCA  based  upon  the  Ph  data  (Table  4-1)  from  the 
received  message.  The  target  position  is  a  function  of  the 
following  independent  variables:  aircraft  position,  aircraft 
heading,  rocket  steering  cursor  azimuth  offset,  and  firing 
crewmember's  LOS.  When  the  actual  time  equals  the  time  of  impact 
of  the  previously  stored  weapon  event  message  or  if  the  time  of 
impact  has  passed,  the  RTCA  results  are  implemented  and  the 
target  notifies  its  status.  Figure  4-3  depicts  the 
geometrically-paired  rocket  engagement. 

Model  Assumptions 

For  the  purpose  of  simplification,  the  rocket  event  model 
employs  certain  assumptions: 
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Aircraft 


Rocket  Geometric  Event 

•  Longbow  Player  ID 

•  Event  Time  (event  end  time) 

•  Weapon  Type  (Rocket-6PD  or  6MP) 

•  Aircraft  Position 

•  Target  Position 

•  Target  Zone 

6PD  for  point  detonating  warhead 
6MP  for  submunition  warhead 

•  Time  of  Impact 

•  Rockets  Fired 

•  Ph:  F(range,  warhead,  rockets  fired) 


Figure  4-3 

Geometrically  Paired  Rocket  Event 

(Adapted  from:  Real  Time  Casualty  Assessment  (RTCA)  for  the 
Longbow  TES  System,  1998) 
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.  all  Longbow  aircraft  perform  identically  if  the  event 
conditions  are  met. 

.  all  environmental  and  human  factors  influences  affecting 
the  engagement  remain  constant. 

The  Ph  for  each  munitions  type  in  the  model  is  purely  an  average 
that  has  been  determined  over  time.  The  constants  implicit  in 
this  average  are  weather  and  atmospheric  conditions,  ballistics, 
crew  experience,  and  proficiency.  A  moderately  experienced 
attack  helicopter  aircrew  realizes  that  the  dispersion  pattern  is 
also  a  function  of  aircraft  accelerations  (natural  and  pilot 
induced)  and  the  effects  of  rotor  downwash  inconsistencies  which 
can  be  influenced  by  the  crew.  Additionally,  asymmetrical 
dispersion  may  be  pronounced  due  to  crosswinds.  The  model 
essentially  rewards  a  substandard  or  inexperienced  crew  with  a 
higher  Ph  while  employing  improper  weapons  techniques . 

Conversely,  it  applies  an  artificially  low  Ph  to  a  crew  that 
adjusts  for  aircraft  inconsistencies.  The  important  point  is 
that  this  model  like  all  others  that  attempt  to  simulate  some 
aspect  of  reality,  has  limitations.  It  is  recommended  that  the 
materiel,  combat,  and  training  developers  work  in  coordination 
with  the  Army  Material  Systems  Analysis  Agency  (AMSAA)  to  focus 
the.  development  efforts  on  weapons  systems  models  that  take  into 
account  the  aircraft  accelerations  and  aerial  ballistic  effects. 
These  would  be  superior  to  the  preliminary  rocket  model  which 
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simply  applies  a  generic  Ph  given  the  launch  constraints  have 
been  met.  When  the  Longbow  TES  System  weapons  models  to  be  used 
for  RTCA  are  made  available,  the  developers  should  look  closely 
at  the  limitations  and  collectively  determine  the  amount  of 
training  fidelity  that  can  be  sacrificed  for  training  benefit. 
The  development  team  must  also  weigh  the  training  fidelity 
against  the  cost  per  unit  improvement  of  model  design  and 
determine  to  what  degree  the  added  fidelity  would  justify  the 
extra  cost. 
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PART  TWO 


Part  Two  provides  the  description  of  the  requirements  for 
Verification  and  Validation,  and  proposed  methodologies  for  V&V 
of  the  Longbow  TES. 
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CHAPTER  5 


VERIFICATION,  VALIDATION  &  ACCREDITATION 


GENERAL 

Verification,  Validation,  and  Accreditation  (W&A)  refers 
to  the  total  process  used  to  ensure  that  the  application  of 
Models  and  Simulations  {M&S)  results  are  appropriate  for  a 
specified  purpose.  The  term  W&A  does  not  refer  to  a  single 
entity,  but  rather  consists  of  three  separate  processes  that 
address  Verification,  Validation,  and  Accreditation  of  M&S 
separately.  Verification  and  Validation  (V&V)  functions  are 
performed  throughout  the  M&S  development  process.  Accreditation 
is  a  subsequent  decision  to  use  M&S  and  the  results  for  a 
particular  application.  The  intent  of  this  Verification  and 
Validation  is  to  measure  the  credibility  and  capability  of  the 
Tactical  Engagement  Simulation  performed  by  the  proposed  Tactical 
Engagement  Simulation  System  for  the  Longbow  Apache.  A 
subsequent  accreditation  will  formally  approve,  conditionally 
approve,  or  disapprove  the  integrated  system  as  an  adequate 
weapons  system  simulator  to  be  used  in  a  representative 
operational  environment. 
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TERMINOLOGY 


The  precise  definitions  of  the  terminology  used  in  the 
intent  statement  of  this  W&A  are  essential  to  understanding  the 
activities  involved,  the  scope  of  these  activities,  and  the 
techniques  used  to  establish  the  value  of  the  TES  System  as  a 
training  tool.  The  definitions  are  consistent  with  those 
currently  used  throughout  the  Department  of  Defense  (DoD)  and 
industry. 

Models  and  Simulation  (M&S) 

Model  According  to  Army  Regulation  5-11,  a  model  is  a  "physical, 
mathematical,  or  otherwise  logical  representation  of  a  system, 
entity,  phenomenon,  or  process"  (DA,  1997b,  16) .  In  the  case  of 
the  Longbow  TES,  the  model  is  the  conceptualization  of  the 
aircraft  weapons  systems  capabilities  converted  into  mathematical 
equations  and  solution  approaches  (algorithms) . 

Simulation  A  simulation  is  the  software  implementation  of  the 
equations  within  the  context  of  a  scenario.  Simulation  is  a 
method  for  implementing  a  model  over  time  (ibid,  17) .  The 
software  used  to  represent  the  actions  and  effects  of  the 
aircraft  weapons  systems  is  the  simulation  in  the  case  of  Longbow 
TES.  The  U.  S.  Army  recognizes  three  categories  of  simulations: 
virtual,  constructive,  and  live.  Virtual  simulation  involves 
networking  a  series  of  simulators  to  support  collective  training 
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on  a  simulated  battlefield.  Constructive  simulation  uses 
networked  computers  to  conduct  wargames  that  focus  on  command  and 
control  decision  making.  Live'simulations  are  combined  arms 
field  training  exercises,  conducted  by  soldiers  employing  real 
and  training  weapons  systems  in  representative  battlefield 
conditions . 

The  Tactical  Engagement  Simulation  System,  composed  of  a 
combination  of  hardware  and  software,  is  a  tool  designed  for  the 
purpose  of  carrying  out  the  effects  of  live  simulation.  It  is  in 
essence,  a  simulator,  intended  to  provide  an  artificial,  but 
realistic  environment  in  which  a  pilot  can  interact  with  certain 
aspects  of  reality.  The  simulator  aims  to  reproduce  those 
aspects  of  reality  essential  to  training  Army  pilots  in  the 
proper  employment  of  weapons  systems  and  contemporary  attack  . 
helicopter  tactics  in  a  representative  operational  environment. 

Verification  Fundamentals 

Verification  focuses  on  the  capability  of  the  Models  and 
Simulations.  According  to  DA  PAM  5-11,  verification  is  the 
process  of  determining  that  a  model  accurately  represents  the 
developer's  conceptual  description  and  specifications,  and  meets 
the  user's  needs  stated  in  the  requirements  document  (DA,  1993, 

26)  . 
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Validation  Fundamentals 


Validation  focuses  on  the  credibility  of  the  Models  and 
Simulations.  According  to  DA  PAM  5-11,  validation  is  "the 
process  of  determining  the  degree  to  which  a  model  is  an  accurate 
representation  of  the  real-world  from  the  perspective  of  the 
intended  uses  of  the  model"  (ibid,  26) .  It  is  important  that 
emphasis  is  placed  on  assessing  the  Tactical  Engagement 
Simulation  in  terms  of  how  it  will  be  used.  This  emphasis 
determines  the  degree  of  detail  that  must  be  represented  for  the 
simulation  to  provide  meaningful  results,  and  the  degree  of 
relationship  with  real-world  phenomena  that  will  be  sufficient  in 
order  to  use  the  TES  with  confidence. 

Validation  aims  to  ensure  that  the  simulation  conforms  to  a 
specified  level  of  fidelity  when  the  TES  system  outputs  are 
compared  to  real-world  weapons  engagements.  Thus,  there  are  two 
stipulations  for  good  validation:  an  understanding  of  the 
intended  use  of  the  model,  and  a  clear  definition  of  the  real- 
world.  Knowledge  of  the  intended  use  outlines  the  requirements 
for  what  needs  to  .be  modeled  and  how  well  those  functions  need  to 
match  the  real-world. 

Accreditation  Fundamentals 

Once  a  simulation  has  been  verified  and  validated  in 
accordance  with  requirements  defined  by  the  intended  application, 
a  determination  of  accreditation  for  the  specified  use  is  made. 
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According  to  DA  PAM  5-11,  "accreditation  is  the  official 
certif ication  that  a  model  or  simulation  is  acceptable  to  be  used 
for  a  specific  purpose"  (ibid,  ~25) .  Accreditation  should  not  be 
considered  an  assumed  conclusion.  It  is  a  decision  that  a 
specific  simulation  can  be  used  for  a  specific  application,  based 
on  evidence  of  suitability  for  the  application.  Neither  is  it 
intended  to  be  a  binary  choice;  credit  or  not  accredit.  The 
accreditation  process  makes  provisions  for  conditional 
accreditation  pending  evidence  of  increased  suitability  for  the 
application.  To  ensure  that  confidence  in  the  TES  is  justified, 
the  V&V  should  ensure  that  the  modeling  assumptions  are  well 
documented,  the  results  produced  by  the  M&S  are  reasonable,  and 
the  correlation  between  the  M&S  behavior  and  real-world  behavior 
is  understood. 

V&V  TASK  SELECTION 

The  Longbow  TES  System  Requirements  Document  defines  the 
functional  requirements,  which  specify  what  the  TES  must  be  able 
to  do:  "... faithfully  replicate  the  weapons  systems  of  the 
Longbow  Apache"  (TRADOC,  1997,  1) .  The  fidelity  requirements, 
are  broadly  defined  by  the  term  faithfully.  How  closely  these 
functions  of  the  TES  correspond  to  the  real-world  will  determine 
the  adequacy  of  the  system  for  use  as  a  training  tool.  The 
accepted  techniques  used  for  V&V  are  generally  grouped  into  four 
categories:  informal,  static,  dynamic,  and  formal.  Informal 
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techniques  to  a  large  degree  depend  on  human  reasoning  and 
subjectivity  without  the  use  of  rigorous  mathematical  concepts. 
Static  techniques  examine  the  accuracy  of  a  static  model  design 
and  its  source  code.  Dynamic  techniques  assess  the  behavior  of 
model  outputs  based  upon  specific  inputs.  Formal  techniques  rely 
heavily  upon  the  use  of  formal  mathematical  proofs  (DMSO,  1996, 
4-1) .  Table  5-1  depicts  the  four  general  V&V  categories  and  some 
commonly  used  techniques  associated  with  each.  The  TES  weapons 
models  (algorithms)  and  the  integrated  TES  System  will  be 
evaluated  through  the  use  of  dynamic  V&V  techniques;  specifically 
functional  testing  or  black-box  testing,  and  field  testing. 
Functional  testing  assesses  the  accuracy  of  input-output 
transformation  (DMSO,  1996,  4-18) .  The  concept  of  functional 
testing  is  depicted  in  Figure  5-1;  the  behavior  of  the  models 
outputs  (specifically  the  prescribed  Ph  associated  with  each 


Table  5-1 

_  V&V  Categories  and  Techniques 


(Adapted  from:  W&A  Recommended  Practices  Guide,  p.  4-2) 


Informal 

Static 

Dynamic 

Formal 

Audit 

Data  Flow 

Analysis 

Functional 

Testing 

Induction 

Inspection 

Fault/Failure 

Analysis 

Field  Testing 

Inference 

Review 

Cause-effect 

analysis 

Product  Testing 

Logical 

Deduction 

Walkthrough 

Structural 

Analysis 

Performance 

Testing 

Inductive  | 

Assertion  | 
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algorithm)  and  the  degree  to  which  the  integrated  TES  System 
replicates  the  real-world,  will  be  assessed  based  on  specific 
inputs  (realistic  operating  conditions) .  Field  testing  is  a 
generic  term  used  to  describe  any  tests  which  are  conducted  by 
placing  M&S  in  operational  scenarios.  Field  testing  is  a  major 
element  of  V&V  activities  conducted  during  the  development  of  new 
weapons  systems  and  military  combat  systems  in  general,  and  is 
regularly  conducted  as  part  of  the  test  and  evaluation  process 
for  DoD  system  acquisition.  Other  methods  in  the  category  of 
dynamic  V&V  techniques  that  are  suitable  for  the  validation  of 
the  Longbow  TES  are  product  testing,  performance  testing,  and 
field  testing.  Product  testing  is  normally  conducted  by  the  M&S 
developer  after  a  piecewise  validation  has  been  performed  on  all 
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of  the  sub-components,  and  before  the  acceptance  testing  is 
performed  by  the  M&S  sponsor  or  proponent  (Schach,  1996) . 
Performance  testing  is  used  to "substantiate  system  performance 
characteristics  (Blanchard,  1990) .  Product  testing  and 
performance  testing  activities  were  scheduled  to  be  completed  by 
the  prime  contractor  and  the  materiel  developer  prior  to  this  V&V 
effort,  but  due  to  system  availability,  were  not  conducted. 

VERIFICATION  METHODOLOGY 

The  proposed  models  verification  methodology  is  a 
rudimentary  form  of  functional  testing.  The  weapons  systems 
models  will  be  run  under  test  data;  their  accurate  execution  will 
be  substantiated  through  attaining  the  representative  Ph  for  each 
engagement  type,  given  the  appropriate  operational  inputs  and 
firing  constraints.  Models  are  typically  evaluated  by  comparing 
their  outputs  to  the  actual  system  outputs  after  both  the  model 
and  the  system  are  run  under  identical  test  data.  Multivariate 
statistical  techniques  are  then  used  to  correlate  the  outputs 
(DMSO,  1996,  4-27).  This  proposed  methodology  relies  on  the  use 
of  weapons  algorithms  created  from  correct  and  accurate 
ballistics  data.  This  precludes  the  need  to  simultaneously 
evaluate  the  Ph  of  the  actual  weapons  systems,  and  the  use  of 
rigorous  statistical  analyses.  The  methodology  employs  the  use 
of  a  "success  template"  and  tracking  sheet  for  each  engagement 
type.  The  success  template  defines  success  as  achieving  the 
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expected  number  of  hits  for  the  particular  engagement  type  under 
the  prescribed  conditions.  The  preliminary  RTCA  rocket 
engagement  from  Chapter  4  is  referenced  for  the  purpose  of 
presenting  an  example  template  and  tracking  sheet.  The 
appropriate  conditions  were  extracted  from  Table  4-1;  (Rocket 
Event  Ph  for  Pair  Fired)  to  derive  the  rocket  event  template 
depicted  as  Table  5-2.  As  an  example,  during  an  engagement 
sequence,  the  model  is  executed  and  the  hit  information  is 
recorded  to  an  output  file  used  for  the  RTCA  process.  The  output 
file  is  examined  to  reveal  the  actual  hit/miss  information.  The 
inputs  (firing  constraints  data  and  the  independent  variables) 
are  extracted  from  the  symbology,  messages,  icons,  and  video  from 
the  Longbow's  on-board  Video  Recording  System.  These  input  data 
and  the  output  are  copied  to  the  rocket  event  tracking  sheet 
(Appendix  D,  Figure  D-l)  maintained  in  the  performance  database. 
The  DataBase  Management  System  (DBMS)  is  explained  in  detail  in 
Appendix  B.  The  output  data  provided  by  the  RTCA  process  will  be 
verified  over  time  to  ensure  that  the  outputs  correlate  with 
their  associated  inputs.  If  the  hit  data  does  not  meet  the 
required  minimum  percentage,  a  causality  investigation  will  be 
initiated.  A  causality  investigation  will  also  be  prompted  if 
the  system  awards  hits  when  the  firing  constraints  are  not  met. 
The  most  realistic  simulation  occurs  during  battles  supported  by 
RTCA;  testing  during  these  events  however,  is  resource  intensive 
due  to  complex  instrumentation  requirements  (DA,  1997c,  6.29). 
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Table  5-2 


Rocket  Event  Success  Template 


|  Independent  Variables 

Required 

Target  Type 

Warhead  Type 

Pair(s)  Fired 

Output  (Ph) 

1 

0.0350 

Hard/ Soft-Skin 

(PD) 

2 

0.0688 

3 

0.1014 

4 

0.1328 

1 

0.1500  j 

Soft-Skin 

(MP) 

2 

0.2775 

3 

0.3859 

4 

0.4780 

The  advantage  of  this  methodology  is  that  the  instrumentation 
requirements  are  minimized.  There  is  no  need  for  additional 
instrumentation  beyond  that  required  for  RTCA.  Considering  the 
complexity  of  the  software  and  hardware  interfaces  of  the 
integrated  TES  System,  this  input-output  correlation  is  an 
effective  means  to  assist  in  detecting  possible  model 
representation  errors  as  a  result  of  message-passing  between  the 
various  sub-modules  of  the  integrated  system.  The  construction 
of  the  success  templates  and  their  associated  tracking  sheets 
were  initiated  but  have  not  been  completed  due  to  the  lack  of 
availability  of  approved  weapons  algorithms  for  the  proposed 
system.  Additionally,  the  data  transmission  protocol  for  the 
RTCA  process  (to  provide  hit/miss  information)  is  not  available. 
It  is  recommended  that  following  the  implementation  of  the 
weapons  algorithms  and  the  development  of  the  RTCA  capability, 
the  Program  Manager  in  coordination  with  Army  research  analysts 
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further  refine  and  implement  the  success  template/tracking 
methodology  for  the  V&V  of  the  integrated  Longbow  TES  System. 

VALIDATION  METHODOLOGY 

The  proposed  validation  methodology  assesses  the  degree  to 
which  the  TES  System  accurately  represents  real-world  weapons 
employment,  actions,  and  effects  from  the  perspective  of  the 
system  operators.  The  methodology  however,  is  not  intended  to 
validate  that  the  demonstrated  (Ph)  of  the  TES  models  match  the 
expectations  of  the  aircrews.  This  correlation  is  implicit  in 
the  selection  of  weapons  algorithms  based  upon  correct  and 
accurate  ballistics  data,  and  ideally  is  a  precondition  for  the 
V&V  of  the  integrated  system.  The  methodology  requires  that 
system  operators  complete  questionnaires  to  rate  the  capability 
of  the  TES  system  to  perform  the  required  simulation.  The  user 
rating  questionnaire  is  presented  as  Appendix  C,  Data  Collection 
Form  1.  A  comparative  rating  scale  is  used  to  compare  specific 
phenomena  in  relation  to  the  Longbow  Apache.  Example  survey 
statements  are  "The  displayed  weapons  symbology  with  the  TES 
System  installed  were  identical  to  the  symbology  presented  during 
normal  operations"  and  "The  weapons  initialization  procedures 
with  the  TES  system  installed  were  identical  to  the  procedures 
during  normal  operations".  The  ratings  are  based  on  a  scale  from 
1  to  5,  where  1  =  strongly  disagree  and  5  =  strongly  agree.  Any 
response  less  than  4  is  considered  a  negative  response.  The 
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comparative  rating  scale  was  selected  over  the  use  of  binary 
choices;  yes  or  no  answers.  This  allows  each  question  to 
encompass  a  broad  scope  and  precludes  a  protracted  survey  where 
its  length  outweighs  its  usefulness  for  gathering  information 
(Fink,  1998,  29) .  For  example,  the  statement  regarding  the 
symbology  incorporates  the  gun,  rockets,  missiles  and  all  other 
ancillary  symbology,  messages,  and  icons  provided  during  a 
tactical  engagement.  If  a  pilot  responds  to  this  statement  with 
any  value  less  than  5,  his  substantiating  comment (s)  may  reveal 
an  anomaly  that  would  not  have  been  disclosed  by  the  use  of  "yes 
or  no"  answers;  the  survey  may  not  have  made  provisions  which 
cover  that  specific  anomaly.  The  questionnaire  responses  will  be 
entered  into  the  performance  database.  Additionally,  data  will 
be  aggregated  from  the  Longbow's  on-board  Video  Recording  System 
(VRS )  to  augment  questionable  system-rating  data  when  necessary. 
The  results  of  each  question  will  be  tabulated;  the  mean  value 
will  be  calculated  among  the  survey  population  and  provided  as 
output.  The  results  of  each  supporting  question  will  then  be 
arithmetically  averaged  to  arrive  at  a  quantitative  measure. 
Implicit  in  this  measurement  approach  is  that  the  numeric  results 
of  each  question  are  identically  weighted.  The  design  of  each 
question  allows  it  to  cover  a  broad  area  without  any  overlap  into 
another  domain.  For  example,  the  statement  referring  to  the 
display  of  symbology  will  not  overlap  with  that  pertaining  to  the 
weapons  initialization  procedures  unless  it  can  be  determined 
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that  incorrectly  displayed  symbology  caused  the  crew  to  make  an 
adjustment  to  the  initialization  procedures.  The  benefit  of  such 
a  performance  measurement  strategy  where  the  criterion  is 
determined  by  the  arithmetic  mean  is  that  it  allows  the  survey 
questions  to  be  modified  anytime  without  making  major  adjustments 
to  the  relative  weights.  The  survey  composition  may  be 
arbitrated  between  the  Program  Manager  and  the  materiel  developer 
prior  to  implementation;  adjustments  can  be  made  quickly  and 
easily.  The  shortcoming  to  such  a  method  is  demonstrated  as 
follows.  The  TES  system  proponent  may  deem  that  the  real  value 
of  the  TES  lies  in  its  ability  to  promote  positive  habit  transfer 
in  weapons  employment.  Consider  a  case  where  the  TES  System 
presents  two  anomalies,  one  which  forces  the  crew  to  alter  the 
way  that  they  must  select  the  weapon  during  an  engagement 
sequence,  and  another  which  manifests  itself  as  the  inability  of 
the  system  to  show  the  rounds  decrement  after  successive 
engagements .  The  proponent  may  view  the  alteration  in  the  weapon 
selection  procedures  by  the  crew  to  be  more  crucial  to  training 
and  thus  be  required  to  have  a  greater  effect  on  the  quantitative 
output.  The  important  point  is,  regardless  of  which  method  is 
chosen,  the  database  outputs  for  each  statement  clearly  depict 
the  operators'  unadjusted  relative  rating  of  the  TES  System. 
Database  output  for  the  pilot' s  questionnaire  using  sample  data 
is  included  as  Appendix  D,  Figure  D-2 . 
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V&V  IN  THE  TES  LIFECYCLE 


The  unavailability  of  the  weapons  algorithms  and  the  lack 
of  RTCA  capability  are  major  limitations  to  the  V&V  of  the 
integrated  TES  System.  The  decision  to  conduct  V&V  of  the 
integrated  system  at  the  current  stage  of  development  was 
premature.  Ideally,  the  V&V  activities  should  have  been 
initiated  during  the  "application  process"  when  the  initial  need 
for  Longbow  TES  was  identified.  The  V&V  activities  should 
continue  throughout  the  acquisition  lifecycle. 

It  is  recommended  that  the  Program  Manager  work  closely 
with  the  materiel  developer  and  actively  conduct  and  manage  V&V 
activities  at  all  stages  of  Longbow  TES  development.  These 
activities  should  include  but  not  be  limited  to: 

•  ensuring  correctness  and  accuracy  of  the  selected  weapons 
algorithms  prior  to  their  implementation, 
focusing  V&V  activities  to  support  the  development  of 
RTCA  using  GPS  geometric  pairing  technology, 
emplacing  formal  procedures  for  configuration  management 
and  documentation  status. 

In  addition,  subsequent  V&V  efforts  should  be  documented  in  the 
Army's  Modeling  and  Simulation  Resource  Repository  (MSRR) .  This 
would  ensure  that  the  materiel,  combat,  and  training  developers 
have  easy  access  to  information  regarding  TES  development,  thus 
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providing  clear  focus  for  future  V&V  efforts.  The  MSRR  can 
potentially  foster  Longbow  TES  development  through  additional 
sources  within  the  M&S  community;  the  repository  promotes 
leveraging  of  M&S  technology  throughout  DoD  by  information 
sharing  and  communication. 
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PART  THREE 


Part  Three  presents  the  development  of  a  fundamental  Test 
Design  Plan  to  quantitatively  and  qualitatively  evaluate  the 
TES  weapons  models  and  the  integrated  TES  System  in  an 
operational  environment. 
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CHAPTER  6 


DEVELOPMENT  OP  THE  V&V 


GENERAL 

An  Initial  Operational  Test  (IOT)  was  selected  as  the 
format  of  the  Verification  and  Validation  for  the  Longbow  TES 
System  to  determine  its  operational  effectiveness  and 
suitability.  The  Army  defines  IOT  as  "a  field  test,  conducted 
under  realistic  operational  conditions,  of  a  production  or 
production-representative  system  for  use  by  typical  users  in 
combat  or  otherwise  deployed"  (DA,  1995,  5-3) .  An  IOT  is 
conducted  any  time  prior  to  the  Milestone  III  development  phase 
of  materiel  systems,  typically  to  support  a  full-rate  production 
decision.  Modeling  this  V&V  plan  as  an  IOT  is  not  intended  to 
circumvent  the  requirements  set  forth  in  AR  73-1  and  other 
applicable  Army  Regulations,  nor  is  it  proposed  in  lieu  of 
specific  tests  required  by  the  Army  prior  to  a  production 
decision.  The  selection  of  an  IOT  format  was  two-fold.  The  IOT 
is  an  Army  recognized  method  which  implies  a  dynamic  V&V 
technique  through  the  use  of  field  testing;  in  the  case  of  TES,  . 
the  IOT  lends  itself  to  model  verification.  Furthermore,  the  IOT 
format  allows  for  additional,  important  non-V&V  data  to  be 
collected  simultaneously.  These  additional  data,  normally 
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collected  during  Operational  Tests  will  assess  the  suitability  of 
the  TES  System.  Where  possible,  these  data  will  be  accepted  as 
valid  OT  data  to  reduce  the  future  OT  requirements,  thus 
decreasing  the  time  required  for  the  acquisition  cycle  and 
conserving  resources.  This  philosophy  is  consistent  with  the 
Army's  Accelerated  Software  Development  Process  (ASDP)  to 
expedite  the  development,  testing,  and  fielding  of  materiel 
systems  with  extensive  embedded  software.  In  the  past, 
traditional  weapon  system  OT  required  the  entire  system  to 
successfully  complete  OT  before  being  fielded.  The  ASDP  allows 
for  incremental  blocks  of  testing  and  subsequent  fielding  of 
portions  of  software  intensive  systems  once  successful  OT  of  a 
representative  sample  has  been  performed  (DA,  1997c,  2.8). 
Performing  the  TES  V&V  as  a  combination  of  DT  (V&V)  and  OT 
supports  the  rapid  prototyping  philosophy  of  the  ASDP. 

Considering  the  complexity  of  the  integrated  TES  System,  rapid 
prototyping  is  useful  in  proving  the  concept  of  RTCA  through  the 
use  of  GPS  geometric  pairing.  Additionally,  due  to  the  lack  of 
explicit  guidance  by  the  TES  Master  Plan,  and  because  the  lack  of 
detailed  requirements  for  future  TES  development,  the  IOT  format 
facilitates  examining  the  interoperability  with  the  future  CTC 
and  homestation  training  infrastructures. 

Development  of  the  V&V  for  the  Longbow  TES  System  was 
accomplished  by  following  the  basic  steps  traditionally  used  in 
the  DoD  materiel  systems  evaluation  process.  The  important 
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points  of  each  step  relative  to  the  Longbow  TES  System  are 
emphasized.  The  basic  steps  include  the  following: 


Step 

Step 

Step 

Step 

Step 

Step 


Step 


1:  Understand  System/Mission/Operating  Environment 

2:  Develop  Operational  Issues 

3:  Determine  Criteria 

4:  Formulate  Measures 

5:  Determine  Data  Elements 

6:  Develop  Evaluation  Plan,  to  Include: 

Define  Test  Events 
Define  Test  Conditions 

Determine  Sample  Sizes  and  Operating  Times 
Develop  a  Data  Management  Scheme 
7:  Review  V&V  Plan  (Steps  2-6)  by  Subject  Matter 
Experts  and  the  Program  Manager 


UNDERSTANDING  SYSTEM/MISSION/OPERATING  ENVIRONMENT 

The  preparation  for  the  V&V  was  initiated  by  a  literature 
review  to  obtain  a  background  history  of  TES,  the  proposed 
employment  and  maintenance  concepts,  and  training  implications. 
Information  regarding  the  Longbow  Apache  and  the  baseline  of  RTCA 
systems  currently  available  for  aviation  Force-on-Force  training 
was  reviewed.  Additionally,  an  on-sight  inspection  of  the 
progress  of  Longbow  TES  was  conducted  at  the  Boeing  Plant 
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facility  and  the  ICE  facility  in  Mesa,  Arizona,  and  at  Fort  Hood, 
Texas . 

ESTABLISHING  ISSUES 

Operational  issues  include  any  aspect  of  the  proposed  TES 
system  capabilities  that  must  be  tested  to  determine  the  system 
effectiveness  or  suitability.  The  issues  collectively  determine 
if  the  system  meets  a  capability  and  to  what  degree.  The 
operational  issues  were  derived  from  the  required  capabilities 
listed  in  the  Longbow  TES  System  Requirements  Document.  As  an 
example,  "Does  the  TES  System  faithfully  simulate  the  actions  and 
effects  of  the  aircraft  weapons  systems?"  is  one  of  the 
operational  issues.  The  required  capabilities  have  been 
formalized  as  issues  and  sub-issues  and  then  arranged  in  a 
"dendritic"  structure.  The  dendritic  technique,  a  procedure 
commonly  used  by  the  Army  for  presenting  operational  issues, 
requires  that  the  system  functions  are  first  listed  and  then 
divided  into  primary  issues.  The  primary  issues  are  subdivided 
into  lower  issues  known  as  test  criteria.  The  process,  depicted 
in  Figure  6-1,  is  continued  until  all  issues  terminate  at  data 
elements.  Data  are  items  which  can  be  collected  without  any 
judgment  on  the  part  of  the  data  collector  and  are  not  predicated 
on  any  previous  tests.  This  subdivision  process  provides  a 
linkage  between  the  data  elements  necessary  to  satisfy  the 
operational  issues,  and  an  easy  reference  to  the  issues  at  later 
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stages  in  test  preparation  and  during  the  evaluation  process. 
Figures  6-2  and  6-3  are  initial  dendritic  diagrams  depicting  the 
required  capabilities  established  by  the  Longbow  TES  System 
Requirements  Document . 

Critical  Issues 

Critical  issues  identified  for  OT  are  typically  those 
issues  which  are  of  primary  importance  to  the  Program  Manager 
(PM)  or  the  decision  making  authority  for  determining  if  the 
system  is  to  continue  at  the  milestone  III  phase  of  acquisition. 
In  the  case  of  this  V&V  effort,  the  critical  issues  verify  and 
validate  the  TES  models  and  the  fidelity  of  the  simulation 


Figure  6-1 
Dendritic  Technique 
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Performance  Capabilities 


Operational  Suitability 


Figure  6-3  (Continued) 


provided  by  the  integrated  TES  System.  The  critical  issues  focus 
on  system  performance  and  mission  effectiveness.  These  issues, 
if  satisfied,  signify  the  TES  models  are  ready  for  the 
accreditation  process.  All  other  issues  are  considered 
Additional  Operational  Issues  (AOIs) . 

Additional  Issues 

The  Additional  Operational  Issues  assess  the  suitability  of 
the  TES  System  and  complement  the  critical  elements.  Information 
regarding  system  suitability  is  not  critical  for  the  Verification 
and  Validation  of  the  weapons  systems  simulation;  the  AOIs  allow 
the  system  proponent  to  steer  the  development  toward  the 
requirements  of  future  TES.  For  instance.  Annex  F  to  OPORD  1-95 
expresses  that  all  future  TES  will  be  required  to  interoperate 
with  a  standardized  After  Action  Review  System  (STAARS)  by  the 
year  2005.  The  STAARS  Operational  Requirements  Document  (ORD) 
provides  advance  planning  information  regarding  the  necessary 
criteria  of  the  proposed  systems,  and  the  STAARS  Handbook 
provides  guidance  concerning  AAR  products.  This  handbook  defines 
the  standard  items  that  the  materiel  developers  will  be  required 
to  incorporate  within  future  TES.  Therefore,  although  the 
functions  of  the  TES  System  Central  Collection  Facility  are  not 
crucial  for  the  V&V,  knowledge  of  the  interoperability  between 
the  CCF  and  the  TES  System  Aircraft  Instrumentation  Package  is 
important  for  the  control  of  future  development.  The  data 
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collected  regarding  suitability  (AOIs)  will  be  flags  used  to 
alert  the  Program  Manager  where  future  interoperability  issues 
may  arise  and  allow  for  resolution  early  enough  in  the  system 
development. 

DETERMINING  TEST  CRITERIA 

Test  criteria  are  expressions  of  the  level  of  performance 
required  of  the  Longbow  TES  System  when  operated  by  typical 
aviation  personnel,  to  demonstrate  effectiveness  or  suitability 
for  given  functions.  For  example,  "the  similarity  between  the 
actions  and  effects  of  the  TES  System  and  the  actual  aircraft 
weapons  systems"  is  one  criterion  used  to  satisfy  the  operational 
issue  regarding  TES  simulation.  The  criteria  have  been  refined 
by  the  judgment  and  experience  of  Subject  Matter  Experts  in  an 
attempt  to  fully  address  and  satisfy  the  issues  (COIs  and  AOIs) . 
Figure  6-4  depicts  the  Critical  Operational  Issues  and  their 
associated  criteria  (collectively  referred  to  as  COIC) .  This 
structure  was  developed  by  refining  the  performance  capabilities 
dendritic  (Figure  6-2) .  The  COIC  are  not  pass/fail  criteria. 
Instead  they  are  intended  to  provide  the  decision  making 
authority  with  a  level  of  credibility  and  capability  of  the  TES 
models  and  the  integrated  system.  The  evidence  will  be  used 
later  in  support  of  the  accreditation  process.  The  COIC  are  few 
in  number  relative  to  the  AOIs  which  is  consistent  with  the 
Army's  current  philosophy  regarding  COIC.  In  the  past,  there  was 
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no  specific  consideration  given  to  the  designation  of  critical 
criteria.  All  were,  therefore,  determined  to  be  critical.  This 
produced  COIC  sets  with  numerous  issues  and  criteria  often 
reflecting  performance  expectations.  Consequently,  operationally 
adequate  systems  had  little  chance  of  satisfying  all  of  these 
expectations  (DA,  1996) .  The  COIC  may  appear  "soft"  at  this 
point.  They  have  been  determined  by  SMEs  to  be  realistic  with 
respect  to  the  maturity  of  the  Longbow  TES  System.  The  COIC 
will  evolve  into  more  "firm"  standards  later  in  the  development 
of  the  system.  A  refined  dendritic  depicted  as  Figure  6-5 
presents  the  Additional  Operational  Issues  and  their  associated 
criteria  (referred  to  as  Complimentary  Measures) . 


Figure  6-4 

Critical  Operational  Issues  and  Criteria 
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Additional  Operational  lianas  (AOI) 
<TES  System  Suitability) 


Figure  6-5 

Additional  Operational  Issues  and  Complimentary  Measures 
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FORMATION  OF  MEASURES  AND  DATA  ELEMENTS 

A  measure  is  a  numeric  element  or  relation  used  to  quantify 
or  describe  the  level. of  performance  or  effectiveness  specified 
by  a  criterion.  Typically  during  Operational  Tests,  measures 
rely  heavily  on  qualitative  analysis.  Observations  by 
experienced  aircrews  are  used  to  formulate  broad  based  opinions 
regarding  TES  System  performance.  For  example,  a  rating  scale  is 
applied  to  questionnaire  responses  that  rank  the  similarity 
between  the  actions  and  effects  of  the  TES  System  and  the  actual 
aircraft  weapons  systems.  This  rating  scale  applies  a 
quantitative  measure  to  qualitative  data  to  satisfy  the 
criterion,  which  in  turn  is  used  to  satisfy  the  operational 
issue.  All  issues  and  measures  have  received  a  complete  analysis 
to  ensure  all  dendritic  paths  end  with  a  meaningful  data 
requirement.  Table  6-1  depicts  the  measures,  associated  data 
requirements,  and  possible  data  sources. 


DEVELOPMENT  OF  EVALUATION  PLAN 
General 

The  development  of  the  Test  Design  Plan  was  accomplished  using 
the  Army's  Operational  Test  Design  Concept  (OTDC) .  The  OTDC 
requires  an  analysis  of  the  data  elements  to  determine  the  types 
of  test  events  and  tactical  scenarios  necessary  to  answer  the 
operational  issues. 
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Table  6-1 


Measures ,  Data  Requirements ,  and  Data  Sources 


Measure  of  Performance/ 
Complimentary  Measure 

Data  Requirements 

Data  Sources 

Percentage  of  kill  and  be- 
killed  messages  sent  to  the 
correct  player (s)  in  near 
real-time 

•  Shooter  ID 

•  Target  ID 

.  Shooter  Location 

•  Target  Location 

.  Time  Message  Sent 

.  Time  message  Received 

•  RTCA  Results 

.  Instrumented 

Source 

User' s  rating  of  the  degree 
to  which  the  TES  System 
provides  for  the  use  of 
accurate  weapons  procedures . 

•  Pilot  Opinion 

•  Pilot  Observation 

•  Cockpit  Displays, 
Tactical  Situation 
Displays  (TSDs) 

.  Symbology 

•  Icons 

•  Messages 

.  VRS  Tapes 

.  SME  Observation 

•  Questionnaire 

User' s  rating  of  the  degree 
to  which  the  TES  System 
delivers  realistic 
simulations  of  flash  and 
smoke  produced  by  the  weapons 
systems . 

•  Pilot  Opinion 

•  Pilot  Observation 

.  VRS  Tapes 

.  SME  Observation 

•  Questionnaire 

Transparency  of  TES  System 
instrumentation  to  operators. 

•  Pilot  Opinion 

•  Pilot  Observation 

.  VRS  Tapes  i 

.  SME  Observation 

.  Questionnaire 

Percentage  of  successful  hits 
using  the  TES  System.  (Ph) 

.  Weapon  System 
.  Munitions  Type 
.  Range 

•  Azimuth 

.  Target  Type 

•  Crewmember  LOS 

•  Rounds  Fired 

•  Rounds  Hit 

.  Instrumented 

Source 

.  VRS  Tapes 

User' s  rating  of  the  degree 
to  which  the  TES  System 
provides  timely  and  accurate 
information  to  support  battle 
management . 

•  Battlestaff  Opinion 

.  Tactical  Icons 

.  Messages 

.  Engagement  Event  No. 

•  Engagement  Time 

•  Receipt  of  RTCA 

Message 

•  Video/Audio  in 

CCF 

Instrumented 

Source  (for 
messages) 

•  Questionnaire 

•  SME  Observation 

User's  rating  of  the  degree 
to  which  the  TES  System 
provides  accurate  information 
to  support  the  preparation 
and  presentation  of  After 

Action  review  (AAR) . 

•  Battlestaff  Opinion 
.  Pilot  Opinion 

•  OC  Opinion 

•  Tactical  Icons 

•  Engagement  Displays 

•  Engagement  Times 

•  Video/Audio 
during  AAR 

•  Questionnaire 

•  SME  Observation 

Mean  Time  To  Repair  (MTTR) . 

•  Repair  Times 

•  Maintenance  Actions 

.  RAM  Data 

Collection 

Procedures 

Operational  Availability  (A<,)  . 

•  Operating/Standby 

Times 

.  Maintenance  Times 

.  RAM  Data 

Collection 

Procedures 
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Table  6-1  (Continued) 


Measure  of  Performance/ 
Complimentary  Measure 

Data  Requirements 

Data  Sources 

Mean  Time  Between  Mission 
Affecting  Failures  (MTBMAF) . 

•  Total  System  Operating 
Time 

.  Mission  Affecting 
Failures 

•  RAM  Data 

Collection 

Procedures 

Reliability. 

.  Operating  Times 

.  Mission  Affecting 

Failures 

.  RAM  Data 

Collection 

Procedures 

Maintenance  Ratio  (MR) . 

.  Maintenance  Times 
.  Operating  Times 

.  RAM  Data 

Collection 

Procedures 

Adequacy  of  support 
equipment  and  Test 

Measurement  and  Diagnostic 
Equipment  (TMDE) . 

.  Reports  of  TMDE  and 

Maintenance  Problems 
due  to  TMDE 

.  Informal  Review 

Checklist 

.  SME  Observation 

Installation/De-installation 

.  Start  Time 
.  Stop  Time 

.  SME  Observation 
.  Stopwatch 
.  Video/Audio 
Recordings 

Assessment  of  the  physical 
characteristics  of  the 

Central  Collection  Facility 
(CCF) . 

.  Physical 

Characteristics 
.  Power  Requirements 

.  Weight/Stability 
•  Heating,  Ventilation, 

Air  Conditioning 
.  Secure  Lighting 
.  Seating  Capacity 

.  Questionnaire 
.  Informal  Review 

Checklist 

.  SME  Observation 

Assessment  of  Human  Factors 
Engineering  (HFE) 
deficiencies 

.  Observation  and  Reports 
of  HFE  Problems 

.  Informal  Review 

Checklist 

•  SME  Observation 

Assessment  of  physical 
characteristics  of  TES  B-Kit 
transit  cases. 

.  Physical 

Characteristics 
.  Carrying  Handles 
•  Size 

Weight 

.  Informal  Review 

Checklist 

.  Test  Observation 

Assessment  of  training 
problems 

.  Observation  and  Reports 
of  Training  Problems 

.  Informal  Review 

Checklist 

.  Test  Observation 

Assessment  of  manpower 
problems 

•  Observation  and  Reports 
of  Manpower  Problems 

.  Informal  Review 

Checklist 

.  Test  Observation 

Assessment  of  personnel 
problems 

.  Tasks  that  Require 
Different  Personnel 
Characteristics  than 
Authorized 

.  Observation  and  Reports 
of  Personnel  Problems 

.  Informal  Review 

Checklist 

.  Test  Observation 

Assessment  of  health 
hazards . 

.  Reports  of  Health 

Hazards 

.  Test  Observation 

Safety  problem  severity. 

•  Reports  of  Safety 

Hazards 

•  Test  Observation 

Assessment  of  soldier 
survivability  problems. 

.  Reports  of  Soldier 

Survivability  Problems . 

•  Test  Observation 
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The  OTDC  identifies  the:  ' 

scenarios  and  the  types  of  events. 

list  of  factors  and  conditions  likely  to  effect  the 
outcomes  of  test  events. 

•  sample  sizes  and  system  operating  times  required  to  control 
the  risks  associated  with  the  anticipated  analyses  (DA, 
1997b,  6.26) . 

The  resulting  Test  Design  Plan  included  as  Appendix  B,  presents 
the  operational  issues  in  detail  to  include  the  scope  of  each 
issue,  supporting  criteria  (or  Complimentary  Measures) ,  basis  for 
the  criteria,  and  the  measures.  In  addition,  the  plan  provides  a 
detailed  description  of  the  data  management  scheme.  Data 
collection  forms  to  support  the  test  plan  are  included  in 
Appendix  C.  This  plan  can  be  modified  as  necessary  and  is 
intended  to  be  used  for  future  Longbow  TES  System  V&V  efforts  if 
required. 

Test  Events 

General  In  an  ideal  world  where  resources  and  schedules  are  not 
concerns,  the  data  requirements  generally  form  the  basis  for 
postulating  the  test  scenarios.  The  decisions  where  and  when  to 
conduct  the  V&V  activities  for  the  integrated  Longbow  TES  System 
were  determined  independently  of  the  development  of  the  formal 
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test  plan.  The  proposed  test  events  are  examined  to  determine 
the  data  elements  that  can  potentially  be  collected.  These  data 
are  compared  to  the  required  data;  the  effects  of  any  differences 
are  compared  to  those  necessary  to  reach  meaningful  conclusions. 
Any  disparity  between  the  required  events  and  the  proposed  events 
which  could  have  an  effect  on  obtaining  defensible  test  results 
is  documented  as  a  test  limitation. 

The  functions,  interfaces,  and  equipment  for  the  proposed 
Longbow  TES  system  were  conceived  under  the  Phase  I  demonstration 
of  CHOICE  at  Fort  Hood,  Texas.  The  TES  System,  intended  to  be  an 
extension  of  the  capabilities  of  CHOICE,  is  being  developed 
simultaneously.  Like  CHOICE,  the  development  of  the  TES  System 
is  to  be  characterized  by  a  rapid-prototyping  paradigm.  The 
CHOICE  System  is  currently  being  used  at  Fort  Hood  by  the  21st 
Cavalry  Brigade,  for  the  purpose  of  conducting  AARs .  Implicit  in 
using  the  system,  executable  pieces  of  CHOICE  are  being 
demonstrated  to  the  21st  Cav  as  the  functions  are  made  available. 
Consequently,  the  unit  provides  the  necessary  feedback  to  refine 
the  system.  The  intent  of  the  prime  contractor  and  the  materiel 
developer  was  to  initiate  the  same  type  of  iterative  development 
process  for  the  TES  System,  beginning  when  the  Longbow  was 
introduced  into  the  Army  inventory. 

As  each  of  the  Army' s  newly  formed  Longbow  equipped 
aviation  battalions  takes  delivery  of  its  aircraft,  the  battalion 
must  progress  through  a  Unit  Training  Program  (UTP) .  The  21st 
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Cav  conducts  the  program  by  training  and  evaluating  Army  AH-64D 
pilots  in  unit  level  mission  essential  tasks.  Upon  completion  of 
the  training,  the  21st  Cav  confirms  the  unit  as  combat-ready. 

The  Army's  first  Longbow  unit,  the  l/227th  Aviation  Battalion, 
was  scheduled  to  go  through  the  Unit  Training  Program  at  Fort 
Hood  from  August  through  October  of  1998.  This  UTP  rotation  was 
to  simultaneously  provide  the  test  events  for  the  Longbow  TES  V&V 
activities.  Table  6-2  depicts  the  events  that  currently 
constitute  the  AH-64D  Unit  Training  Program. 

Aerial  Gunnery  Exercise  Prior  to  completing  the  Table  VIII  crew 

qualification  aerial  gunnery  exercise,  each  aircrew  receives 

* 

practice,  known  as  a  Table  VII.  This  practice  table  will 
simultaneously  serve  as  a  pilot  test  to  familiarize  the  test 
control  personnel  and  data  collectors  with  the  general  conduct  of 
the  V&V  procedures  to  ensure  seamless  execution  of  their 
responsibilities.  During  the  exercise,  each  aircraft  and  target 
will  be  instrumented  with  the  TES  system  and  employed 
individually  in  a  series  of  controlled  engagement  scenarios. 

This  gunnery  table  adds  realism  through  the  employment  of  target 
acquisition  and  engagement  opportunities,  both  moving  and 
stationary,  that  an  aircrew  would  likely  encounter  in  an  isolated 
(single-ship)  combat  situation.  Data  collected  during  this  event 
will  focus  on  the  TES  system  performance  issues.  The  gunnery 
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Table  6-2 


AH-64D  Unit  Training  Program  Events  List 

(Source:  21st  Cavalry  Brigade,  Fort  Hood,  TX.,  1998) 


MISSION 

EVENT 

ECHELON 

SUPPORTED 

THREAT 

DURATION 

PER 

MISSION 

(hr) 

ITERATIONS 

TOTAL 

TIME 

(hr) 

Aerial  Gunnery 
(Crew  Qual . ) 

Crew 

Heavy  Armor, 
Soft-skin  (14) 

a  1.0 

24 

24 

Movement  to 
Contact/Screen 

Company 

Company 

Size  Element 

a  2.0 

3 

6 

Deliberate 

Attack 

Company 

Company 

Size  Element 

a  2.0 

3 

6 

Hasty  Attack 

Company 

Company 

Size  Element 

a  2.0 

3 

6 

Screen/Flank 

Battalion 

Battalion 

Size  Element 

«  5.0 

2 

10 

Movement  to 
Contact 

Battalion 

Battalion 

Size  Element 

a  5.0 

2 

10 

Guard 

Battalion 

Battalion 

Size  Element 

a  5.0 

2 

10 

event  provides  for  the  use  of  surveyed  firing  pads,  firing  lanes, 
and  controlled  target  arrays  which  will  allow  for  instrumented 
collection  of  truth  data  regarding  aircraft  and  threat  target 
location  to  evaluate  the  GPS  geometric  pairing  accuracy  used  for 
RTCA.  Data  collectors  will  gather  data  for  mission  performance, 
scenarios.  Reliability,  Availability,  and  Maintainability  (RAM), 
manpower  and  personnel  integration  (MANPRINT) ,  and  installation 
and  de-installation.  Table  6-3  provides  a  detailed  list  of  the 
target  engagement  scenarios  used  in  the  AH-64D  Table  VII/VIII 
gunnery  during  the  UTP. 


66 


d 

0 

• 

P 

P 

(0 

0 

W 

■H 

43 

- 

rH 

0 

Em 

m 

0 

43 

c 

P 

U 

G 

0 

G 

0) 

d) 

•rt 

d 

a 

£ 

43 

0 

P 

a> 

43 

P 

P 

0 

Oi 

P 

•rl 

S 

-rl 

u 

cd 

•rl 

3 

Eh 

G 

Oi 

Oi 

< 

& 

G 

w 

0 

0 

0) 

d 

P 

0 

P 

43 

Q> 

G 

43 

P 

rH 

P 

•H 

P 

0 

<d 

cd 

O 

0 

G 

P 

•rl 

a 

G 

0) 

•H 

d 

0 

-rl 

• 

•H 

0 

0) 

G 

d 

d 

> 

w 

G 

d 

•rl 

0 

<u 

•H 

w 

•rl 

0 

P 

p 

(3 

d 

cd 

P 

G 

o 

0 

•H 

G 

-rl 

-rl 

o 

-rl 

0 

•H 

0) 

4H 

rH 

0 

rH 

4J 

G 

P 

G 

rH 

-rl 

rH 

P 

p 

P 

O 

rH 

P 

cd 

0 

0 

0 

u 

<d 

0 

P 

a 

d 

u 

d 

0 

0 

p 

0 

<u 

0 

-H 

0 

p 

o 

P 

4-> 

> 

a 

(3 

0) 

4J 

•H 

G 

0 

0 

rH 

G 

d 

O 

0 

43 

£ 

a 

•rl 

G 

•H 

p 

P 

(P 

£ 

o 

H 

P 

p 

Oi 

o 

a 

U 

0 

G 

id 

u 

0 

G 

•rH  ■ 

Oi 

d 

rH 

0 

g 

0 

G 

CO 

0 

P 

(U 

4J 

cd 

G 

0 

O 

P 

o 

P 

u 

1 — 1 

(U 

0 

-H 

P 

u 

rH 

£ 

0) 

P 

0 

d 

o 

0 

•rH 

£ 

•rl 

a 

0 

P 

•rl 

W 

0 

P 

>1 

p 

P 

0 

p 

O 

rH 

o 

d 

a 

a 

0 

-H 

LW 

0) 

£ 

£ 

rH 

P 

P 

3 

G 

•rl 

0 

0 

w 

0 

£ 

0 

0 

0 

0 

1 — 1 

•rH 

© 

P 

0 

e 

rH 

X 

5 

0 

0 

0 

■H 

id 

cd 

4-) 

MH 

•rl 

U 

P 

£ 

0 

0 

£ 

rQ 

d 

0 

G 

rH 

3 

d 

0 

a 

(d 

B 

G 

>1 

P 

p 

P 

P 

•rl 

cd 

rH 

U 

p 

0 

0 

X 

i — 1 

P 

0 

G 

P 

id 

£ 

0 

d 

0 

£ 

p 

o 

0 

rH 

0) 

£ 

•H 

d 

0 

i — 1 

P 

0) 

•H 

P 

0 

•H 

0 

43 

G 

U 

0 

d 

P 

•rl 

0 

43 

-rl 

0 

£ 

4J 

> 

u 

0 

P 

rH 

H 

£ 

0 

(U 

0) 

rH 

P 

Eh 

■H 

> 

> 

G 

•rl 

0 

P 

o 

cd 

•H 

3 

a 

0 

43 

> 

0 

0) 

0 

0 

p 

G 

o 

G 

i — 1 

£ 

4J 

a 

-H 

w 

•H 

rH 

G 

43 

Eh 

0 

•rl 

0 

■rl 

0 

P 

Oi 

4J 

p 

O 

43 

-rH 

0 

0 

a 

P 

5 

a 

P 

3 

0 

0 

>1 

d 

P 

0 

0 

MH 

P 

0 

G 

0 

P 

Jh 

G 

43 

0 

43 

G 

O 

d 

0 

P 

-P 

0 

0 

£ 

3 

£ 

P 

P 

0 

0 

0 

0 

0 

c 

U 

Oi 

P 

P 

P 

Oi 

cu 

P 

cd 

P 

0 

P 

0 

£ 

d 

Oi 

0 

0 

Oi 

<u 

0) 

G 

G 

0 

G 

•  G 

Oi 

d 

0 

0 

43 

0 

0 

id 

4-) 

Oi 

0) 

rH 

4J 

P 

i — I 

c 

P 

td 

0 

O 

0 

0 

w 

<d 

P 

P 

P 

P 

P 

d 

E 

£ 

d 

■p 

w 

•H 

a 

•rl 

© 

P 

> 

0 

p^ 

0 

> 

Oi 

G 

•H 

5 

5 

•rH 

P 

*H 

d 

0 

0 

0 

d 

aj 

o 

G 

p 

p 

p 

G 

Eh 

Oj 

H 

u 

0 

u 

H 

rH 

CN 

m 

ID 

Company  and  Battalion  Maneuver  Exercises  The  live  maneuver 
exercises  will  be  used  to  evaluate  the  Tactical  Engagement 
Simulation  in  an  operationally -representative  combined  arms 
environment.  With  large  numbers  of  instrumented  aircraft  and 
ground  vehicles  participating  in  the  Force-on-Force  exercises, 
there  exists  the  possibility  of  data  transmission  latency  that 
may  not  be  encountered  or  evident  during  the  Table  VII  gunnery 
event.  The  effects  of  data  latency  translate  into  a  target 
management  issue  for  the  Longbow  that  could  impact  the  timing  and 
outcomes,  and  thus  the  realism  of  engagement  simulation.  The 
effects  of  data  latency  will  be  examined  during  the  maneuver 
exercises.  These  exercises  will  serve  as  a  secondary  source  of 
system  performance  and  effectiveness  data  used  to  augment  the 
gunnery  event  data.  In  addition  to  providing  data  for  Tactical 
Engagement  Simulation,  these  events  afford  qualitative  and 
quantitative  data  on  how  well  the  TES  System  provides  timely  and 
accurate  information  to  support  battle  management  by  examining 
the  interoperability  with  the  Central  Collection  Facility.  The 
maneuver  exercises  will  also  support  data  collection  for  RAM, 
MANPRINT,  and  installation  and  de-installation  operations. 

Tactical  Context  of  the  Events  During  the  gunnery  Table  VII  and 
the  maneuver  exercises,  the  TES  System  will  be  employed  against 
various  tactical  echelons  of  wheeled  and  tracked  vehicles  (single 
vehicle,  team,  platoon,  company,  and  battalion) .  The  target  sets 
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are  operationally  realistic  and  their  activities  will  be 
consistent  with  the  appropriate  threat  tactical  doctrine.  The 
level  of  conflict  intensity  witl  be  low  for  the  gunnery  exercise, 
and  will  range  from  low  to  high-intensity  for  the  maneuver 
exercises . 

Test  Factors  and  Conditions 

Test  factors,  sometimes  called  variables,  are  those  items 
which  could  possibly  influence  the  system  performance  during  the 
test.  Factors  are  generally  divided  into  three  categories; 
friendly,  enemy,  and  environmental.  Friendly  factors  include: 
equipment,  doctrine,  mission,  crew,  training,  logistic  support, 
morale,  and  leadership.  Some  examples  of  enemy  factors  are: 
target  types,  countermeasures,  and  mobility.  Environmental 
factors  are  items  such  as  terrain,  weather,  and  climate.  These 
factors  are  further  defined  as  either  controllable  or 
uncontrollable.  Controlled  factors  are  those  that  can  be  varied 
or  held  constant  during  the  test  to  produce  the  desired 
conditions.  Uncontrolled  factors  also  may  produce  different 
conditions  during  the  test,  however  there  is  little  or  no  control 
over  them.  Examples  of  uncontrolled  factors  are  pilot  or 
maintainer  motivation  and  weather.  The  controlled  variables  can 
be  an  asset  to  the  V&V  because  they  potentially  add  variations  to 
a  particular  test  and  should  produce  additional  data. 

Recognition  of  the  uncontrolled  factors  is  important  since  they 
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lead  to  test  limitations.  An  example  of  a  test  limitation  is  the 
presence  of  adverse  weather  during  authorized  range  test  period; 
reduced  visibility  during  allocated  test  time  may  restrict  flight 
crews  to  perform  limited  range  weapons  engagements. 

Realizing  that  there  is  practically  no  limit  to  the  number  of 
variables  associated  with  a  test,  a  list  was  constructed  which 
contains  only  the  major  factors  which  may  significantly  impact 
the  test.  The  factors  were  then  associated  with  the  conditions 
under  which  they  should  be  observed.  The  appropriate  conditions 
are  representative  of  those  under  which  the  TES  system  is  to  be 
employed  as  stated  in  the  mission  profile  section  of  the 
Requirements  Document.  Emphasis  was  placed  on  selecting 
systematically  variable  conditions  for  only  those  factors 
necessary  to  test  the  system.  The  major  event  factors, 
conditions,  and  variable  control  methods  are  listed  in  table  6-4. 

In  developing  a  test  design,  it  is  desired  to  vary  each 
factor  that  can  possibly  be  varied,  to  collect  as  much  supporting 
data  as  possible.  The  drawback  to  such  a  philosophy  is  that  by 
introducing  more  combinations  of  variables  into  a  test,  the 
magnitude  of  the  test  effort  quickly  becomes  unwieldy. 

Referencing  Table  6-4,  there  are  nine  major  factors  that  can  be 
systematically  varied  during  a  proposed  test.  Holding  the  range 
constant,  there  are  12,960  distinct  combinations  of  the  variable 
factors.  One  realistic  combination  could  be  defined  by  the 
following  engagement:  aircraft  at  night,  part  of  a  company  level 
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Table  6-4 

Event  Factors,  Controls,  and  Conditions 


FACTOR 

CONTROL 

CONDITIONS 

Echelon  supported  by  the 
TES  System 

Systematically  Varied 

Single-Ship,  Platoon, 
Company,  Battalion 

Interoperability 

Held  Constant 

MILES /AGES, 

Central  Collection 

Facility  (CCF) 

Mission 

Systematically  Varied 

Gunnery,  Attack, 
Reconnaissance,  Security 

Mission  Duration 

Systematically  Varied 

Gunnery  Event:  »lhr 

Company  Mission:  *2hr 
Battalion  Mission:  «5hr 

System  Operating  Status 

Uncontrolled 

Fully  Mission  Capable, 

Part.  Mission  Capable, 

Not  Mission  Capable 

Aircraft  Activities 

Systematically  Varied 

Hovering 

Moving 

Weapon  Systems 

■ 

Systematically  Varied 

30mm  gun 

2.75"  rockets 

Hellfire  Missile 

Crewmember  Line-of-Sight 
(LOS) 

Systematically  Varied 

IDM-TADS/TADS 

IHADSS 

Cooperative 

Target  Types 

Systematically  Varied 

Light  Armor,  Heavy  Armor, 
Wheeled/Track  Vehicles,  ; 

Troops,  ADA  I 

Target  Activities 

Systematically  Varied 

Moving,  Stationary 

Target  Range 

Systematically  Varied 

500m  to  8000m 

Personnel 

Held  Constant 

MOS  152XX  Aviator 

MOS  68X  Maintainer 

Organization 

Held  Constant 

Two  152XX  Per  TES  System 
Aircraft  Instrumentation 
Package 

Training 

Held  Constant 

Provided  by  System 
Contractor 

Maintenance  and  Logistics 
Support 

. 

Held  Constant 

Operator  and  Unit  Level- 
68X,  Above  Unit  Level- 
System  Contractor 

Electromagnetic 
Interference  Environment 
(EMI) 

Uncontrolled 

As  Occurs 

Weather  Conditions 

Uncontrolled 

As  Occurs 

Light  Conditions 

Systematically  Varied 

Day,  Night 

attack  mission,  engaging  a  stationary,  heavily  armored  vehicle  at 
2,000  meters  with  3  pairs  of  (6MP)  rockets.  Some  of  the 
conditions  can  be  grouped  together  thus  reducing  the  complexity 
of  the  V&V  effort.  For  instance,  the  function  "Target  Type"  can 
be  varied  in  six  ways:  heavy  armor,  light  armor,  wheeled 
vehicle,  tracked  vehicle,  air  defense,  and  troops.  The 
preliminary  TES  rocket  algorithm  in  Chapter  4  does  not 
differentiate  between  vehicle  types;  it  merely  defines  a  target 
in  one  of  two  ways,  heavy  armor  (hard-skin)  or  not  heavy  armor 
(soft-skin) .  Consequently,  this  function  only  needs  to  be  varied 
in  two  ways.  Since  the  TES  System  should  not  perform  any 
different  during  the  day  than  it  does  in  darkness,  the  function 
"Light  Conditions"  no  longer  needs  to  be  an  independent  variable. 
Additionally,  the  range  can  be  defined  by  a  limited  number  of 
discrete  increments,  i.e.  1000m  range  intervals.  After 
consolidating  these  few  conditions,  the  possible  combinations 
have  dropped  significantly.  However,  care  should  be  taken  when 
reducing  the  number  of  conditions.  For  instance,  it  could  be 
argued  that  "Mission  Duration"  does  not  need  to  be  an  independent 
factor,  since  the  behavior  of  the  TES  models  should  be 
independent  of  system  operating  time.  However,  longer  continuous 
operation  time  is  associated  with  the  battalion  Force-on-Force 
missions  which  involve  large  numbers  of  instrumented  aircraft  and 
ground  vehicles.  The  expected  increase  in  the  frequency  and 
amount  of  dataflow  between  aircraft  and  ground  targets  examines 
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the  GPS  geometric  pairing  capability  of  the  TES  System  under 
extreme  workload  conditions.  The  increase  in  the  system  workload 
may  create  congestion  which  translates  into  the  inability  to 
correctly  pair  targets  and  arbitrate  hits.  Additionally,  longer 
continuous  operating  times  provide  data  to  substantiate  system 
reliability.  Table  6-5  depicts  the  factors,  controls,  and 
conditions  after  combining  appropriate  conditions. 

Engagement  Opportunities  and  System  Operating  Time 

The  gunnery  event  provides  for  approximately  336  total 
engagement  opportunities,  however  the  total  number  of  engagements 
can  not  be  reasonably  estimated  for  the  maneuver  exercises.  In 
all  cases  of  the  live  maneuver  exercises,  the  initial  battle 
conditions  are  determined  (systematically  varied)  before  the 
start  of  each  event,  however  the  individual  engagement 
opportunities  develop  tactically  and  can  not  be  guaranteed.  As 
an  example,  the  movement-to-contact  mission  often  results  in  a 
meeting  engagement  where  forces  enter  into  conflict  with  each 
other  by  chance  rather  than  by  design.  The  gunnery  event  and  the 
maneuver  exercises  provide  for  approximately  24  hours  and  48 
hours  of  event  time  respectively,  which  collectively  afford 
approximately  468  hours  of  total  system  operating  time.  Table  6- 
6  depicts  a  proposed  TES  System  usage  schedule  which  maximizes 
the  amount  of  event  time  for  each  of  the  ten  (10)  TES  System  AIPs 
designated  for  the  V&V  effort. 
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Table  6-5 


Event  Factors ,  Controls ,  and  Combined  Conditions 


FACTOR 

CONTROL 

CONDITIONS 

Echelon  supported  by  the 
TES  System 

Systematically  Varied 

Single-Ship,  Other  Than 
Single-Ship 

Interoperability 

Held  Constant 

MILES /AGES, 

Central  Collection 
Facility  (CCF) 

Mission 

Systematically  Varied 

Gunnery,  All  Other 

Mission  Types 

Mission  Duration 

Systematically  Varied 

Gunnery  Event :  «lhr 

All  Other  Missions:  >lhr 

System  Operating  Status 

Uncontrolled 

Fully  Mission  Capable, 
Part.  Mission  Capable, 

Not  Mission  Capable 

Aircraft  Activities 

Systematically  Varied 

Hovering 

Moving 

Weapon  Systems 

Systematically  Varied 

30mm  gun 

2.75"  rockets 

Hellfire  Missile 

Crewmember  Line-of-Sight 
(LOS) 

Systematically  Varied 

IDM-TADS/TADS 

IHADSS 

Cooperative 

Target  Types 

Systematically  Varied 

Light  Armor,  Heavy  Armor 

Target  Activities 

Systematically  Varied 

Moving,  Stationary 

Target  Range 

Systematically  Varied 

Discrete  1000m  Increments 
from  1000m  to  8000m 

Personnel 

Held  Constant 

MOS  152XX  Aviator 

MOS  68X  Maintainer 

Organization 

Held  Constant 

Two  152XX  Per  TES  System 
Aircraft  Instrumentation 
Package 

Training 

Held  Constant 

Provided  by  System 
Contractor 

Maintenance  and  Logistics 
Support 

Held  Constant 

Operator  and  Unit  Level- 
68X,  Above  Unit  Level- 
System  Contractor 

Electromagnetic 
Interference  Environment 
(EMI) 

Uncontrolled 

As  Occurs 

Weather  Conditions 

Uncontrolled 

As  Occurs 

Table  6-6 


TES  System  Usage  Schedule 


TES  System 
AIP  No. 

Operating  Time  ~  hours  j 

Gunnery 

Event 

Company 

Exercises 

Battalion 

Exercises 

Time  per 
AIP 

1  1 

2 

15 

30 

47 

i  2 

2 

15 

30 

47 

3 

2 

15 

30 

47 

4 

2 

15 

30 

47  | 

5 

2 

15 

30 

47 

6 

2 

15 

30 

47  j 

7 

2 

15 

30 

47 

8 

2 

15 

30 

47  | 

|  9 

2 

15 

30 

47 

10 

6 

9 

30 

45 

|  Total  Operating  Time  (TOT) 

468 

Test  Limitations  By  inspection  of  Table  6-5;  (Combined 
Conditions) ,  the  proposed  test  events  can  potentially  generate 
the  required  data  elements  listed  in  Table  6-1;  (Measures, 
Requirements,  and  Data  Sources) .  These  events  however  can  not 
provide  the  number  of  engagement  opportunities  necessary  to 
determine  the  fidelity  of  the  weapons  models.  Furthermore,  the 
corresponding  total  system  operating  time  is  marginally 
sufficient  to  establish  the  confidence  of  the  additional  OT  data; 
specifically  the  reliability  data.  Table  6-7  depicts  the  number 
of  engagement  opportunities  required  to  establish  an  80  percent 
confidence  level  that  the  probabilities-of-hit  provided  as  output 
by  the  TES  System  are  reliable  data.  This  table  was  constructed 
based  on  the  data  provided  by  the  rocket  event  success  template 
combined  with  the  rocket  engagement  opportunities  afforded  by  the 
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Table  6-7 


Sample  Size  Required  to  Establish  an  80%  Confidence  Level 


ENGAGEMENT 

TARGET 

m 

PAIR(S) 

PROBABILITY 

REQUIRED 

NUMBER 

TYPE 

FIRED 

OF  HIT  (Ph) 

SAMPLE  SIZE 

1 

0.0350 

22 

5 

Soft-skin 

(PD) 

2 

0.0688 

42 

3 

0.1014 

60 

4 

0.1328 

76 

1 

0.1500 

84 

11 

Soft-skin 

(MP) 

2 

0.2775 

132 

3 

0.3859 

155 

1 

0.1500 

84 

12 

Soft-skin 

(MP) 

2 

0.2775 

132 

3 

0.3859 

155 

test  events  listed  in  Table  6-3;  (Table  VII/VIII  Target  Engagement 
Group  Summary) .  The  following  formula  was  used  to  determine  the 
sample  size  for  each  engagement  type: 

z2Ph(l-^i) 

e2 

where: 

n  =  the  required  sample  size 

z  =  value  for  an  80  percent  confidence  level,  using  a 
normal  distribution 

Ph  =  expected  probability  of  hit  (Ph)  from  template 
e  =  margin  of  error  (±5  percent) 

The  values  for  the  confidence  level  and  margin  of  error  are 
reasonable  values  for  OT  and  were  arbitrarily  chosen  for  the 
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purpose  of  illustration  (DA,  1997c) .  Estimates  of  sample  sizes 
are  useful  for  controlled  experiments;  Neville  (1964)  provides  a 
detailed  explanation  for  determining  sample  sizes  for  the  design 
of  experiments.  Due  to  the  large  number  of  combinations  of  test 
conditions  which  may  occur  during  Operational  Testing,  the 
conditions  necessary  for  precise  sample  sizing  are  rarely 
satisfied.  However,  fundamental  statistical  assumptions  and 
quasi-experimental  design  provide  reasonable  sample  size 
approximations  for  planning  of  OT  (DA,  1997c,  6.31).  Assuming  24 
flight  crews  participate  in  the  Table  VII/VIII  gunnery  event, 
engagement  No.  5  would  provide  at  most  24  engagement 
opportunities  out  of  76  required  to  establish  an  80  percent 
confidence  level.  Additionally,  assuming  that  6MP  warheads  are 
used  for  engagements  11  and  12,  these  engagements  would  provide 
at  most  48  opportunities  out  of  155  required.  The  proposed  test 
events  only  guarantee  336  total  engagement  opportunities;  the 
rocket  events  would  collectively  require  231  to  substantiate  only 
the  rocket  algorithm.  Furthermore,  if  an  aircrew  shoots  less 
than  the  pairs  of  rockets  allocated  for  any  of  these  engagements, 
the  expected  Ph  is  no  longer  the  same  so  the  data  can  not  be 
incorporated  into  the  analysis.  Therefore,  based  on  the 
preliminary  rocket  algorithm,  the  proposed  test  events  can  not  be 
expected  to  generate  enough  data  to  adequately  evaluate  all  of 
the  weapons  systems  models.  It  is  recommended  that  following  the 
implementation  of  the  weapons  systems  models  to  be  used  for  RTCA, 
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the  Program  Manager  in  coordination  with  Army  research  analysts 
determine  the  required  weapons  engagements  to  substantiate  the 
input-output  transformation  of'the  TES  algorithms  using  the 
success  template  methodology.  The  resulting  determination  should 
then  be  used  to  define  the  specific  V&V  events  and  scenarios. 
Furthermore,  it  is  recommended  that  the  V&V  activities  of  the 
integrated  TES  System  be  performed  at  the  unit  homestations  and 
CTCs  in  addition  to  Fort  Hood  to  expedite  the  acquisition  of  the 
TES  System.  Boeing  had  scheduled  186  hours  of 
product/performance  testing  preceding  the  V&V  effort;  it  is 
recommended  that  they  focus  their  data  collection  efforts  on 
weapons  engagements  that  reveal  the  statistical  need  for  larger 
sample  sizes,  and  also  on  engagements  that  are  not  likely  to  be 
performed  during  the  maneuver  exercises.  This  includes 
engagement  scenarios  at  the  maximum  limits  of  the  weapons  systems 
i.e.  4,000m  gun  engagements  and  8,000m  Hellfire  missile 
engagements .  These  data  could  be  used  to  augment  the  Army' s  V&V 
data  as  necessary.  In  the  worst  case,  if  fiscal  constraints 
preclude  testing  beyond  one  UTP  rotation,  the  decision  to 
extrapolate  the  data  for  the  purpose  of  model  accreditation  would 
incur  risk. 

The  total  system  operating  time  provided  by  the  proposed 
test  events  to  establish  confidence  in  the  OT  data  is  marginal. 
The  mean  time  between  failure,  defined  as  the  Mean  Time  Between 
Mission  Affecting  Failure  (MTBMAF)  by  the  TES  Sytem  Requirements 
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Document,  is  one  of  the  basic  and  more  important  measures 
included  in  formal  tests  and  demonstrations  of  repairable  systems 
(Blanchard,  1990,  105) .  Figure  6-6  was  developed  to  facilitate 
determining  the  maximum  allowable  number  of  Mission  Affecting 
Failures  (MAF)  during  the  prescribed  test  period.  This  chart, 
commonly  used  by  DoD  for  reliability  testing  was  constructed 
using  the  following  assumptions: 

1.  10%  risk  to  the  materiel  developer  that  the  TES  System 
with  a  true  reliability  which  meets  the  required  value  (MTBMAF  > 
50  hours)  will  be  rejected  as  not  meeting  the  requirement. 

2.  10%  risk  to  the  system  proponent  that  the  TES  System 
with  a  true  reliability  which  does  not  meet  the  lower  test  limit 
(MTBMAF  <  40  hours)  will  be  accepted  as  meeting  the  requirement. 

40  hours  represents  the  lowest  MTBMAF  that  is  considered  to  be 
acceptable  based  on  actual  test  results.  Figure  6-6  reveals  that 
if  the  TES  System .experiences  at  least  one  MAF  during  the 
prescribed  test  period  depicted  in  Table  6-6;  (TES  System  Usage 
Schedule) ,  the  corresponding  amount  of  system  operating  time  is 
not  sufficient  to  accurately  determine  system  reliability.  The 
intersection  of  Total  Operating  Time  and  Total  Mission  Affecting 
Failures  must  fall  below  the  "accept"  line.  Therefore,  in  the 
case  that  these  V&V  activities  are  used  to  simultaneously  support 


79 


Figure  6-6 

TES  System  Reliability  Test  Plan 

OT,  one  Mission  Affecting  Failure  implies  that  the  decision 
making  authority  can  not  advocate  a  subsequent  production 
decision  in  good  faith  without  further  testing.  Blanchard  (1990) 
provides  a  detailed  explanation  regarding  the  construction  and 
use  of  the  reliability  test  plan  (382)  .  Again,  it  is  recommended 
that  the  V&V  activities  of  the  integrated  TES  System  be  performed 
at  other  training  sites  in  addition  to  Fort  Hood  to  increase  the 
amount  of  TES  system  operating  time . 

Data  Management 

The  data  management  process  condenses  and  formats  the  V&V 
data  for  analysis,  documentation,  and  storage.  A  preliminary 
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"level-four"  database  structure  has  been  developed  for  the  V&V 
effort.  The  Army  defines  level-four  data  as  tables,  charts,  bar 
graphs,  relative  frequencies  of  judgments  or  qualitative  data, 
and  data  summaries  showing  totals,  means,  and  elapsed  times.  A 
detailed  description  of  the  DataBase  Management  System  (DBMS)  is 
included  in  Appendix  B.  Database  outputs  using  sample  data  are 
included  as  Appendix  D. 


REVIEW  EVALUATION  PLAN 

Coordination  and  integration  between  the  participants  of 
the  V&V  effort,  and  the  materiel,  training,  and  combat  developers 
is  essential  to  the  success  of  the  V&V  processes  in  the 
acquisition  of  the  Longbow  TES  System.  It  is  necessary  for  the 
evaluators  and  the  combat  and  training  developers  to  review  the 
V&V  plan;  the  output  of  these  reviews  are  necessary  to  further 
refine  the  plan  prior  to  the  V&V  of  the  integrated  TES  System. 

The  development  of  the  V&V  plan  is  intended  to  be  an  iterative 
process.  The  tasks  associated  with  the  development  of  a  sound 
V&V  plan  for  a  complex  system  such  as  the  Longbow  TES  System, 
span  a  large  spectrum.  Consequently,  the  planning  of  future  V&V 
efforts  would  be  well  served  by  an  organization  that  has  broad 
experience  in  applicable  specialized  areas  such  as  software 
engineering,  modeling  and  simulation,  systems  engineering, 
operations  research,  and  test  and  evaluation.  It  is  recommended 
that  the  Program  Executive  Office  (PEO) ,  Longbow  Apache,  identify 
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the  U.  S.  Army  Test  and  Evaluation  Command  (ATEC)  to  work  closely 
with  the  University  of  Tennessee  Space  Institute  on  the  planning 
of  future  V&V  efforts.  It  is  recommended  that  ATEC  review  the 
proposed  V&V  methodologies  and  the  Test  Design  Plan,  and  adapt 
them  as  necessary  for  future  testing  of  the  Longbow  TES  System. 
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CHAPTER  7 


CONCLUSIONS  AND  RECOMMENDATIONS 


CONCLUSIONS 

The  proponent  for  the  Longbow  TES,  the  U.  S.  Army  Aviation 
Center  (USAAVNC) ,  U.  S.  Army  Training  and  Doctrine  Command 
(TRADOC)  is  examining  the  impact  of  Tactical  Engagement 
Simulation  development  on  combat  aviation  training. 

Consequently,  the  Program  Executive  Office  (PEO) ,  Longbow  Apache, 
directed  a  Verification  and  Validation  (V&V)  to  examine  the 
proposed  Longbow  TES  System  at  its  current  stage  of  development. 

A  fundamental  verification  methodology  was  developed  to  evaluate 
the  fidelity  of  the  Longbow  TES  weapons  algorithms  in  a  realistic 
operational  environment.  This  method  employs  a  rudimentary  form 
of  functional  testing;  verifying  the  model  outputs  (the  expected 
weapons  systems  Ph)  based  upon  the  inputs  (operational 
conditions) .  The  advantage  of  this  methodology  is  that  the 
instrumentation  requirements  are  minimized;  there  is  no  need  for 
additional  instrumentation  beyond  that  required  for  RTCA.  The 
disadvantage  of  this  methodology  is  the  requirement  for  large 
sample  sizes  needed  to  adequately  verify  the  model  outputs  (Ph 
data) .  A  validation  methodology  was  developed  to  determine  the 
fidelity  of  the  simulation  provided  by  the  integrated  system. 
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This  methodology  validates  system  performance  by  applying 
quantitative  measures  to  principally  qualitative  system  outputs. 

A  test  plan  was  developed  that'incorporates  these  Verification 
and  Validation  techniques  with  the  simultaneous  collection  of 
valuable  Operational  Test  data.  This  promotes  rapid-prototyping, 
thus  reducing  future  OT  requirements,  and  conserving  resources  by 
expediting  the  TES  System  acquisition  process.  The  test  plan 
incorporates  a  data  management  scheme  and  a  level-four  database 
structure  designed  specifically  for  evaluation  of  the  Longbow  TES 
System.  The  V&V  efforts  conducted  by  the  University  of  Tennessee 
Space  Institute  (UTSI)  flight  systems  research  group  resulted  in 
the  following  conclusions: 

1.  The  decision  to  conduct  V&V  of  the  integrated  TES  system  at 
its  current  stage  of  development  was  premature;  the  installed 
system  was  not  available  for  operation.  The  lack  of 
finalized  weapons  systems  algorithms  and  lack  of  RTCA 
capability  (GPS  geometric  pairing)  were  major  limitations  to 
the  Verification  and  Validation  of  the  integrated  system. 
Consequently,  the  proposed  V&V  methodologies  and  Test  Design 
Plan  were  not  implemented. 

2.  V&V  activities  should  not  to  be  conducted  as  a  one  time 
process  during  system  development.  Ideally;  the  V&V 
activities  should  have  been  initiated  during  the  application 
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process  when  the  initial  need  for  Longbow  TES  was  identified. 
A  strong  V&V  plan  which  promotes  efficient  system 
development,  incorporates  Verification  and  Validation 
techniques  beginning  at  the  application  process  and 
continuing  throughout  the  acquisition  lifecycle. 

3.  The  development  of  the  Longbow  TES  System  and  V&V  efforts  are 
not  being  tracked  or  documented  in  a  Resource  Repository. 

4.  Based  upon  the  data  from  a  preliminary  weapon  system  model, 
the  events  scheduled  for  these  V&V  activities  would  not  have 
been  adequate  to  verify  the  weapons  systems  models  to  a 
reasonable  (80  percent)  confidence  level. 

5.  In  the  event  that  these  V&V  activities  were  to  simultaneously 
support  OT,  the  amount  of  total  system  operating  time  would 
have  been  marginally  sufficient  to  support  a  production 
decision,  based  on  reliability  data.  One  (1)  Mission 
Affecting  Failure  during  the  proposed  test  period  would 
prevent  the  decision  making  authority  from  making  a 
production  decision  in  good  faith  without  further  testing. 


RECOMMENDATIONS 

1.  It  is  recommended  that  the  Program  Manager  work  closely  with 
the  materiel  developer  and  actively  conduct  and  manage  V&V 
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activities  at  all  stages  of  Longbow  TES  development.  These 
activities  should  include  but  not  be  limited  to: 

.  ensuring  correctness  and  accuracy  of  the  selected 
weapons  algorithms  prior  to  their  implementation. 

.  focusing  the  V&V  activities  to  support  the 

development  of  RTCA  using  GPS  geometric  pairing 
technology.  V&V  efforts  of  the  integrated  system 
should  be  continued  only  after  the  RTCA  technology 
is  developed. 

•  emplacing  formal  procedures  for  configuration 
management  and  documentation  status . 

2.  Subsequent  V&V  efforts  should  be  documented  in  the  Army's 
Modeling  and  Simulation  Resource  Repository  (MSRR) .  This 
would  ensure  easy  access  to  information  regarding  TES 
development,  thus  providing  clear  focus  for  future  V&V 
efforts.  The  MSRR  can  potentially  foster  TES  development 
(RTCA  capability)  through  additional  sources  within  the  M&S 
community. 

3.  The  materiel,  combat,  and  training  developers  should  work  in 
coordination  with  the  Army  Material  Systems  Analysis  Agency 
(AMSAA)  to  focus  the  development  efforts  on  weapons  systems 
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models  that  take  into  account  aircraft  accelerations  and 


aerial  ballistic  effects,  to  provide  greater  training  value. 

4.  Army  research  analysts  should  use  these  proposed  V&V 
methodologies  as  a  starting  point  and  further  refine  and 
implement  them  for  V&V  of  the  integrated  Longbow  TES  System. 

5.  Army  test  and  evaluation  personnel  should  determine  the 
required  weapons  engagements  to  substantiate  the  input-output 
transformation  of  the  TES  algorithms  using  the  success 
template  methodology.  The  resulting  determination  should 
subsequently  be  used  to  define  the  specific  V&V  events  and 
scenarios . 

6.  Performing  V&V  activities  of  the  integrated  TES  System  at  the 
unit  homestations  and  CTCs  in  addition  to  Fort  Hood  would 
expedite  the  acquisition  of  the  TES  System.  Furthermore,  it 
is  recommended  that  Army  test  and  evaluation  personnel  work 
closely  with  Boeing  to  guide  the  focus  of  the  prime 
contractor' s  data  collection  efforts  during 
product/performance  testing.  This  would  potentially 
facilitate  augmenting  the  Army' s  V&V  data  with  contractor 
data  if  necessary. 
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7.  The  planning  of  future  V&V  efforts  would  be  well  served  by  an 
organization  that  has  broad  experience  in  applicable 
specialized  areas  such  as  software  engineering,  modeling  and 
simulation,  systems  engineering,  operations  research,  and 
test  and  evaluation.  It  is  recommended  that  the  Program 
Executive  Office  (PEO) ,  Longbow  Apache,  identify  the  U.  S. 
Army  Test  and  Evaluation  Command  (ATEC)  to  work  closely  with 
the  University  of  Tennessee  Space  Institute  on  the  planning 
and  implementation  of  future  V&V  efforts. 

8 .  It  is  recommended  that  ATEC  review  the  proposed  V&V 
methodologies  and  the  Test  Design  Plan,  and  adapt  them  as 
necessary  for  future  testing  of  the  Longbow  TES  System. 
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APPENDIX  A 


DETAILED  SYSTEMS  DESCRIPTIONS 


WEAPONS  DESCRIPTIONS 
Fire  Control  Radar  (FCR) 

The  Fire  Control  Radar  (FCR)  is  an  integrated  millimeter 
wave  radar  system  with  a  mast-mounted  transmitter  and  receiver. 
The  FCR  detects,  locates,  and  classifies  ground  and  airborne 
targets  and  provides  terrain  profile  mapping  for  operating  in 
limited  visibility  conditions.  The  FCR,  in  combination  with  the 
RF  Hellfire  missile,  provides  a  fire-and-forget  capability.  A 
mast-mounted  Radio  Frequency  Interferometer  (RFI)  is  incorporated 
to  provide  threat  emitter  warning,  azimuth,  and  direction 
finding.  RFI  detected  emitter  signals  may  be  correlated  with  FCR 
target  data  to  derive  the  identity  and  location  of  a  particular 
threat.  The  target  information  is  sent  to  the  Weapons  Processor 
for  the  determination  of  target  location  coordinates  and  is  used 
to  direct  the  weapons  against  potential  targets.  FCR  target 
information  is  displayed  to  the  aircrew  on  the  FCR  page  and  the 
Tactical  Situation  Display  (TSD)  page.  The  FCR  page  targeting 
format  may  also  be  displayed  on  the  Copilot  Gunner's  (CPG) 
displays.  In  addition,  the  information  is  processed  for  the 
generation  of  FCR  target  symbology  for  overlay  onto  the  Target 
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Acquisition  and  Designation  Sensor  (TADS) /Pilot' s  Night  Vision 
Sensor  (PNVS)  video. 

AGM-114  Hellfire  Missile  System  Description 

AGM-114  Hellfire  Missile  (HELicopter-Launched  FIRE-and- 
forget  missile)  is  the  primary  weapon  system  of  the  Longbow 
Apache,  used  for  the  anti-armor  role.  The  Longbow  is  capable  of 
employing  two  variants  of  this  missile,  the  AGM-114A/C/K  Semi- 
Active  Laser  (SAL)  guided  missile,  and  a  Radio  Frequency  (RF) 
guided  version.  The  Hellfire  missile  can  be  launched  by  the 
pilot  or  the  Copilot/Gunner.  There  are  four  available  SAL 
missile  modes  and  one  RF  mode.  All  of  these  missile  modes  may  be 
designated  autonomously  or  handed-off  from  another  aircraft 
equipped  with  with  a  compatible  digital  information  source. 

AGM-114A/C/K  Semi -Active  Laser  (SAL)  guided  missile  Following  a 
launch,  this  missile  tracks  reflected,  coded  pulse  laser 
radiation  from  a  target  which  has  been  illuminated  by  the 
aircraft  that  launched  the  missile  or  a  separate  laser 
designator.  The  missile  must  be  able  to  see  the  reflected  laser 
light  from  the  time  of  launch  until  it  reaches  the  target.  It 
features  an  Improved  Low-Visibility  (ILV)  detection  autopilot  and 
a  low-smoke  motor  to  reduce  detection.  The  warhead  is  an  8 
kilogram  shaped-charge  with  a  copper  liner.  The  Hellfire  missile 
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travels  at  a  speed  of  Mach  1.4  and  has  a  maximum  range  of  between 


5-8Km. 


AGM-114K  Radar  Frequency  (RF)  guided  missile  The  AGM-114K 
Hellfire  II  features  a  digital  autopilot  and  improved  warhead; 
the  copper  liner  of  the  shaped-charge  warhead  has  been  replaced 
by  a  molybdenum  steel  liner.  Hellfire  II  includes  a  tandem 
warhead  intended  to  defeat  reactive  armor  with  two  charges,  a 
minor  initial  explosion  followed  by  the  main  shaped-charge 
warhead.  The  range  of  the  Hellfire  II  is  in  excess  of  8Km. 

M140  Aerial  Rocket  System  <ARS) 

The  Folding-Fin  Aerial  Rocket  (FFAR)  is  an  area-effect 
weapon  for  the  Apache  Longbow.  This  unguided  rocket  measures 
70mm  in  diameter  and  is  characterized  by  a  set  of  three  wings 
which  fold  around  the  body  of  a  MK66  rocket  motor.  Upon  exit 
from  the  launcher,  the  fins  spring  outward  to  aid  in  stability. 
Additionally,  the  rocket  nozzles  are  scarfed  at  an  angle  to  add  a 
spin  to  the  rocket  during  flight  for  added  stability.  The 
wingspan  is  186mm  when  deployed.  The  maximum  range  is 
approximately  7.5Km.  The  rockets  may  be  employed  by  either 
crewmember  or  cooperatively.  There  are  a  variety  of  warheads 
which  can  be  attached  to  the  MK66  rocket  motor  to  be  used  for 
anti-materiel,  anti-personnel,  and  suppression  missions.  The 
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available  warheads  are  characterized  by  either  penetration  or 
time  delay  detonation.  The  available  warheads  are: 

.  Point-Detonating  (PD)  or  penetration  fuze 
•  airburst-range  with  a  settable  fuze  using  Multi-Purpose 
Sub-Munitions  (MP) . 

M230E1  Chain  Gun 

The  M230E1  Chain  gun  is  located  beneath  the  forward 
fuselage,  directly  below  the  Copilot/Gunner ' s  position.  The  gun 
is  hydraulically  steered  and  has  an  electrical  firing  mechanism. 
To  reduce  jamming,  the  ammunition  feed  uses  an  electrically- 
driven  one-piece  chain  to  feed  the  linkless  shells  into  the  gun. 
Ammunition  travels  down  the  starboard  side  of  the  feed  chute 
while  the  spent  shell  casings  are  carried  up  the  port  side  to  be 
returned  to  the  magazine.  The  rate  of  fire  for  the  M230  is  600- 
650  rounds  per  minute.  Each  round  takes  approximately  2  seconds 
to  travel  1000m  and  12  seconds  to  travel  3000m. 

TES  SYSTEM  DESCRIPTION 
A-Kit 

Weapons  Processor  (WP)  The  Weapons  Processor  is  the  MIL  STD  1553 
Databus  controller.  It  schedules  message  traffic  on  the  weapons 
bus  between  the  remote  terminals  of  the  TES  System.  Software 
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functions  include  weapons  inventory,  TES  System  initialization, 
weapons  emulation,  and  eye-safe  laser  range  processing. 

System  Processor  (SP)  The  System  Processor  software  controls  the 
additional  bus  messages  that  the  TES  System  generates  between  the 
WP  and  Display  Processor  (DP) ,  and  the  DP  and  the  Communications 
Interface  Unit  (CIU) .  When  the  TES  System  is  installed,  the  "TES 
mode"  message  is  sent  from  the  WP  to  the  SP,  and  additional  data 
is  sent  from  the  SP  to  the  DP  to  drive  TES  System  displays.  The 
SP  simulates  all  arm  functions  when  the  TES  system  is  installed. 

Display  Processor  (DP)  TES  System  displays  are  to  be  added  to 
the  DP.  Additional  displays  include  advisory  messages  from  RTCA 
results  and  player  status . 

Communications  Interface  Unit  (CIU)  The  CIU  provides  audible 
real-time  feedback  for  RTCA.  Audio  alert  enunciation  exists  in 
the  CIU.  Sound  effects  for  the  gun,  rockets,  and  missiles  are 
provided  by  the  CIU. 

Radio  Frequency  Interferometer  (RFI)  The  RFI  permits  detection 
and  identification  of  the  simulated  threat  emitters  used  at  the 
CTCs . 
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Radar  Warning  Receiver  A  data  module  for  the  APR-39  Radar 
Warning  Receiver  will  be  integrated  which  permits  identification 
of  the  simulated  threat  emitters  at  the  CTCs. 

B-Kit  Description 

Figure  A-l  depicts  B-Kit  components  with  transit  case. 

From  left  to  right;  SMODIM  mounting  assembly,  GPS  antenna, 
SMODIM,  and  DCI . 

SMODIM  The  SMODIM  is  the  heart  of  the  TES  System  Aircraft 
Instrumentation  Package.  Mounted  in  the  aircraft  survival  kit 


Figure  A-l 

TES  System  B-Kit  Components  with  Transit  Case 
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bay  depicted  in  Figure  A-2,  the  SMODIM  includes  an  embedded  Data 
Recorder  and  performs  MILES  decoder  functions .  The  SMODIM 
communicates  with  the  aircraft-WP  as  a  remote  terminal.  It 
obtains  position  information  from  the  aircraft  Embedded  GPS/INS 
and  provides  this  information  to  the  range  telemetry  system  or 
the  CTC  range  system  through  the  DCI  during  the  conduct  of  live 
maneuver  exercises.  The  SMODIM  coordinates  weapons  inventory 
status  with  the  WP  and  receives  laser  inputs  from  the  AVR-2A 
laser  Detector  and  performs  threat  probability  of  hit  (Ph)  and 
probability  of  kill  (Pk)  evaluation.  The  laser  inputs  are  sorted 
and  filtered  for  false  messages,  assessed,  and  verified.  This 
permits  determination  of  cumulative  hit  or  miss  information  and 
resultant  damage.  The  SMODIM  performs  the  "footprint" 
calculations  that  support  the  Longbow  Hellfire  and  rocket 
geometric  pairing  as  well  as  the  30mm  gun  and  SAL  Hellfire 
targeting.  The  installed  software  will  also  provide  MILES/AGES 
30mm  gun  and  SAL  Hellfire  geometric  pairing  if  the  training 
center  infrastructure  supports  that  means  of  engagement.  The 
SMODIM  also  records  defined  event  and  mission  data. 
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Figure  A- 2 

Smart  On-Board  Data  Interface  Module  (SMODIM) 


Data  Communications  Interface  (DCI)  The  DCI  communicates 
aircraft  position  and  event  data  from  the  SMODIM  to  the  aircraft 
Embedded  GPS/INS  (EGI)  and  receives  commands  from  the  CTC 
communications  network.  The  DCI  antenna  is  depicted  in  Figure  A- 
3  installed  underneath  and  aft  on  the  tailboom.  Note  the  GPS 
antenna  installed  at  the  intersection  of  the  aft  fuselage  and  the 
tailboom.  Also  pictured  is  the  AVR-2A  laser  receiver  attached  to 
the  side  of  the  tailboom. 
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Figure  A-3 

Data  Communications  Interface  (DCI)  Antenna 
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Indicator  Control  Unit  (ICU)  The  Indicator  Control  Unit  (ICU)  is 


located  in  the  TES  System  Training  Missile.  The  ICU  controls 
power,  BIT,  and  the  commands  to  the  Aircraft  Kill  Indicator  (AKI) 
and  FLASHWESS.  It  acts  as  a  remote  terminal  on  Multiplex  Bus 
(MUX)  No.  3.  The  ICU  performs  self-test  and  reports 
AKI/FLASHWESS  BIT  status  to  the  Weapons  Processor. 

TES  System  Training  Missile  (TTM)  The  TTM  is  a  Hellfire  missile 
body  containing  the  ICU,  FLASHWESS,  and  associated  cabling.  The 
TTM  interfaces  with  the  aircraft  via  a  wing  pylon  1760  connector. 
The  TES  System  Training  Missile  is  depicted  in  Figure  A-4.  Note 
the  FLASHWESS  recessed  in  the  seekerhead  and  the  Aircraft  Kill 
Indicator  (blue  light)  on  the  missile  body. 


Figure  A-4 

TES  System  Training  Missile  (TTM) 
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Laser  Rangefinder/Designator  (LRFD)  The  LRFD  is  an  electronic 
assembly  used  in  place  of  the  aircraft  Laser  Transceiver  Unit  and 
Laser  Tracker  Receiver  located~in  the  TADS  turret.  The  LRFD 
interfaces  electrically  with  the  aircraft  via  the  TLIA. 

Training  Laser  Interface  Adapter  (TLIA)  The  TLIA  provides  a  two- 
way  interface  between  the  aircraft  MIL  STD  1553  data  bus,  two 
training  lasers,  and  a  Flash  Weapons  Effects  Signature  Simulator 
(FLASHWESS) . 

30mm  Gun  Laser  The  gun  laser  installs  on  the  gun  carriage  to 

preclude  interference  with  the  lower  Wire  Strike  Protection 

<• 

System  (WSPS)  blade  mounted  on  the  gun  carriage.  It  receives 
power  and  signals  from  the  TLIA  and  transmits  Line-of-Sight 
MILES/AGES  event  data  to  simulate  the  effects  of  the  30mm  gun. 
Figure  A-5  depicts  the  LRFD,  TLIA,  30mm  gun  laser,  and  associated 
cabling. 

Central  Collection  Facility  (CCF)  Van  The  CCF  van  includes  two 
Pentium  300MHz  computer  systems,  a  54-inch  color  display,  a 
communications  interface,  a  self-erecting  antenna  mast,  and  an 
independent  power  supply/generator .  The  CCF  van  is  completely 
mobile  and  field  supportable.  It  provides  commanders  the  ability 
to  monitor  Longbow  training  engagements  in  near  real-time  and  the 
ability  to  conduct  After  Action  Reviews  and  briefings. 
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Figure  A- 5 

30mm  Gun  Simulation  Components 


The  engagement  data  are  stored  and  can  be  replayed  at  the  single 
unit  or  multiple  player  levels.  The  CCF  also  hosts  the  master 
TES  System  instrumentation  package  processor  that  provides  for 
data  storage,  system  analysis,  and  report  output.  The  TES  System 
CCF  van  (Figure  A-6) ,  initially  fielded  under  the  Phase  I  of 
CHOICE,  will  be  upgraded  incrementally  to  expand  the  multimedia 
capabilities  required  for  future  TES.  Real  time  command  net 
recording  will  be  provided  to  augment  AAR  capability;  this 
includes  the  added  capability  of  recording  both  video  and  audio 
during  debriefs. 
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Figure  A- 6 


Central  Collection  Facility  (CCF) 


A  third  computer  will  be  added  to  act  as  a  server  to  enhance  the 
system' s  data  capturing  capability  and  to  off-load  the  graphics 
manipulation  to  the  workstation.  The  CCF  workstation  is  depicted 
as  Figure  A-7. 

Target  Instrumentation  Kits  The  35  target  instrumentation  kits 
include  eye-safe,  laser  based  weapons  simulators,  a 
communications  interface,  and  a  ground  unit  data  recorder.  These 
components  are  capable  of  being  mounted  on  Government  Furnished' 
Vehicles  (GFV)  using  vehicle  power  supply.  Utilizing  a  standard 
interface  to  Government  Furnished  Equipment  Weapons  Effect 
Simulation  (GFE  WES)  kit  hardware,  the  targets  will 


Figure  A- 7 

Central  Collection  Facility  (CCF)  Workstation 


have  a  laser  based  shoot-back  capability  against  aircraft 
equipped  with  the  TES  System. 

Ground  Repeater  Units  Four  ground  repeater  units  are  specialized 
instrumentation  kits  configured  to  provide  expanded  training  area 
coverage  to  support  the  TES  System  data  communications  scheme.  . 
The  ground  repeater  units  are  comprised  of  a  mobile 
communications  interface  and  a  rechargeable  power  supply.  The 
portable  ground  repeater  units  do  not  have  a  weapons  simulator 
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capability,  but  are  operable  with  either  vehicle  or 
Instrumentation  Power  Supply  (IPS) . 

Training  Data  Playback  Units  (TDPU)  Two  Training  Data  Playback 
Units  provide  field  support,  operation,  data  replay,  and  source 
data  collection  functions  for  the  deployed  TES  System 
Instrumentation  Package.  The  TDPU  is  capable  of  accepting  data 
from  the  Longbow  flight  data  recorder,  allowing  data  files  from 
multiple  players  or  targets  to  be  saved  and/or  merged,  and 
viewed. 
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APPENDIX  B 


TEST  DESIGN  PLAN 


OPERATIONAL  ISSUES  AND  CRITERIA 
Critical  Operational  Issue  (COI)  1 

Does  the  TES  System  provide  a  means  of  Real  Time  Casualty 
Assessment  (RTCA)  capability  for  Force-on-Force  collective 
training  exercises? 

Scope  This  issue  examines  the  capability  of  the  Longbow  TES 
System  to  provide  a  means  of  (RTCA)  in  near  real-time  using  GPS 
geometric  pairing.  Single-ship  gunnery  tables  will  be  conducted 
and  company  and  battalion  size  maneuver  exercises  will  be  played 
in  a  representative  combined  arms  environment  with  the  TES  System 
installed.  The  system  will  be  operated  and  maintained  by 
qualified  soldiers  and  contractor  support. 

Criterion  1.1  The  Longbow  TES  System  will  pass  and  receive  90% 
of  kill  and  be-killed  messages  to  the  correct  player  (s),  to 
include  a  status  notification  in  near  real-time  (<  5  seconds) 
during  Force-on-Force  training  exercises 

Basis  for  Criteria  Criterion  1.1  is  derived  from  the  Longbow  TES 
System  Requirements  Document,  Sections  1  and  2. 

Measure  of  Performance  (MOP)  1.1  The  measure (s)  used  to  evaluate 
this  criterion  will  be  determined  pending  development  of  the  RTCA 
capability  using  GPS  geometric  pairing  technology. 


Critical  Operational  Issue  (COI)  2 

Does  the  TES  faithfully  simulate  the  actions  and  the  effects  of 
the  aircraft  weapons  systems? 

Scope  This  issue  examines  the  capability  of  the  Longbow  TES 
System  to  simulate  the  actions  and  effects  of  the  aircraft 
weapons  systems.  This  capability  includes  weapons  initialization 
and  selection  procedures,  the  portrayal  of  weapons  symbology  (to 
include  messages)  and  weapon  status  information,  and  the 
capability  of  achieving  the  probability  of  hit  (Ph)  dictated  by 
the  appropriate  weapons  systems  models.  Furthermore,  this  issue 
evaluates  the  weapons  range  capabilities  to  include  portrayal  of 
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reduced  weapons  capabilities  commensurate  with  adverse  weather 
conditions.  Single-ship  gunnery  tables  will  be  conducted,  and 
company  and  battalion  size  maneuver  exercises  will  be  played  in  a 
representative  combined  arms  environment  with  the  TES  installed. 
The  Longbow  TES  will  be  operated  and  maintained  by  qualified 
soldiers  and  contractor  support. 

Criterion  2.1  Similarity  between  Longbow  weapons  procedures  and 
TES  System  weapons  procedures.  Evaluation  Criteria:  No  Go,  <  4. 

Basis  of  Criterion  Criterion  2.1  is  derived  from  the  Longbow  TES 
System  Requirements  Document,  Sections  1  and  2. 

Measure  of  Performance  2.1  System  operator  ratings  of  the  degree 
to  which  the  TES  System  provides  for  the  use  of  accurate  weapons 
employment  procedures.  These  include  weapons  initialization 
procedures  and  sequence,  weapons  selection,  weapon  status 
information,  appropriate  symbology,  and  weapons  range 
capabilities  during  engagements. 

MOP  Methodology  System  operators  will  complete  questionnaires  to 
rate  the  capability  of  the  TES  system  to  perform  the  tasks 
identified  in  the  criterion.  Data  will  be  aggregated  from 
questionnaires  and  aircraft  VRS  tapes.  Upon  completion  of  the 
test  event,  the  test  control  personnel  and  Subject  Matter  Experts 
will  review  the  data  and  enter  questionnaire  responses  into  the 
performance  database.  The  test  control  personnel  will  assess  the 
information  received  from  the  video  recordings  to  collect  and 
verify  missing  or  questionable  data.  The  test  control  personnel 
will  render  opinions  by  the  application  of  military  judgment  to 
augment  the  system  rating  questionnaire  data  when  necessary. 

Mean  values  will  be  calculated  and  provided  as  output. 

Criterion  2.2  The  degree  of  realism  provided  by  the  flash  and 
smoke  system  during  weapons  system  simulation.  Evaluation 
Criteria:  No  Go,  <4. 

Basis  for  Criteria  Criterion  2.2  is  derived  from  the  Longbow  TES 
System  Requirements  Document,  Section  2. 

Measure  of  Performance  2.2  System  operator  ratings  of  the  degree 
to  which  the  TES  System  Aircraft  Instrumentation  Package  delivers 
realistic  simulations  of  the  flash  and  smoke  produced  by  the 
actual  weapons  systems. 

MOP  Methodology  Event  player  personnel  will  complete 
questionnaires  to  rate  the  system  capability  to  perform  the  tasks 
listed  in  the  criterion.  The  Measure  of  Performance  methodology, 
data  collection  method,  data  aggregation  and  reduction  methods, 
and  database  output  will  be  similar  to  Criterion  2.1. 
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Criterion  2.3  Transparency  of  TES  System  instrumentation  to 
operators.  Evaluation  Criteria:  No  Go,  <  4. 

Basis  for  Criterion  Criterion  2.2  is  derived  from  the  Longbow 
TES  System  Requirements  Document,  Sections  1  and  2. 

Measure  of  Performance  2.3  System  operator  ratings  regarding  the 
transparency  of  the  TES  System  Aircraft  Instrumentation  Package 
to  the  aircrew. 

MOP  Methodology  The  TES  System  Aircraft  Instrumentation  Package 
must  not  enhance  or  detract  from  the  normal  perceptions  of  the 
aircrew  regarding  the  sensors,  weapons  systems,  or  cockpit 
environment.  Additionally,  the  TES  System  components  must  not 
degrade  the  operational  characteristics  of  the  prime  system  or 
host  weapons  systems  or  interfere  with  crew  duties.  System 
operators  will  complete  questionnaires  to  rate  the  system 
capability  to  remain  effectively  transparent  to  the  aircrew.  The 
Measure  of  Performance  methodology,  data  collection  method,  data 
aggregation  and  reduction  methods,  and  database  output  will  be 
similar  to  Criterion  2.1. 

Criterion  2.4  Percentage  of  successful  hits  (Ph)  using  the  TES 
System.  Evaluation  Criteria:  No  Go,  variable  based  on  weapons 
engagement  type . 

Basis  for  Criterion  Criterion  2 . 2  was  derived  from  the  Longbow 
TES  System  Requirements  Document,  Sections  1  and  2,  requiring  the 
TES  System  to  provide  the  same  accuracy  as  the  aircraft  weapons 
systems . 

Measure  of  Performance  (MOP)  2.4  Percentage  of  successful  hits 
using  the  TES  System. 

MOP  Methodology  The  demonstrated  percentage-of-hits  shall  be  the 
same  as  the  expected  probability  of  hit  (Ph)  for  the  TES  weapons 
models  (rocket,  gun,  or  missile  simulation) .  The  output  (Ph)  of 
the  TES  model  will  be  verified  using  the  success 
template/tracking  methodology  given  specific  inputs  (operational 
conditions  and  firing  constraints) .  Data  will  be  aggregated  from 
VRS  tapes  and  RTCA  instrumentation. 

Additional  Operational  Issue  (AOI)  3 

Does  the  TES  System  provide  timely  arid  accurate  information  to 
support  battle  management? 

Scope  This  issue  assesses  the  ability  of  the  TES  System  to 
provide  timely  and  accurate  information  regarding  the  disposition 
of  friendly  and  enemy  forces  in  near  real-time,  to  affect 
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probable  courses  of  action  and  schemes  of  maneuver.  This  issue 
investigates  how  well  the  interface  between  the  TES  System 
Aircraft  Instrumentation  Package  and  the  CCF  supports  the 
tactical  decision  making  efforts  of  the  battlestaff.  Company  and 
battalion  size  maneuver  exercises  will  be  played  in  a 
representative  combined  arms  environment  with  the  TES  System 
installed.  The  battle  staff  will  monitor  the  exercises  from  the 
Central  Collection  Facility  (CCF).  The  console  operations  in  the 
CCF  will  be  performed  by  contractor  support. 

Complimentary  Measure  (CM)  3.1  Battlestaff  member  ratings 
regarding  the  ability  of  the  TES  System  to  provide  timely  and 
accurate  information  to  support  battle  management? 

Basis  for  Measure  This  measure  is  determined  by  the  TES  Master 
Plan,  Section  5-4,  regarding  the  implications  of  future  TES 
requirements,  and  the  HTI-ORD,  Section  1.  (3),  regarding  Exercise 
Management . 

CM  Methodology  Battle  management  will  be  answered  via  the  use  of 
rating  questionnaires,  administered  to  the  battlestaff 
representing  the  various  echelons  supported  by  the  TES  System. 

The  battlestaff  shall  monitor  the  exercise  from  the  Central 
Collection  Facility  and  answer  questions  about  how  well  the  TES 
System  Aircraft  Instrumentation  Package/CCF  interface  supports 
their  tactical  decision  making  efforts.  Data  for  this  measure 
will  be  collected  from  video  recordings  of  the  CCF  operator 
console  screens  and  video  and  audio  recordings  of  battlestaff 
actions  and  conversations  inside  the  CCF.  Upon  completion  of 
each  test  event,  the  test  control  personnel  and  Subject  Matter 
Experts  will  review  the  data  and  enter  questionnaire  responses 
into  the  performance  database.  The  test  control  personnel  will 
assess  the  information  received  from  the  video  recordings  to 
collect  and  verify  missing  or  questionable  data  and  will  render 
opinions  by  the  application  of  military  judgment  to  augment  the 
rating  questionnaire  data  when  necessary.  Mean  values  will  be 
calculated  and  provided  as  output. 


Additional  Operational  Issue  (AOI)  4 

Does  the  TES  System  provide  accurate  data  for  the  preparation  and 
presentation  of  training  performance  feedback  (AAR  Capability) ? 

Scope  This  issue  assesses  the  ability  of  the  TES  System  to 
to  provide  accurate,  readily  interpreted  data  to  be  used  in  the 
preparation  and  presentation  of  an  After  Action  Review  (AAR) . 
Company  and  battalion  size  maneuver  exercises  will  be  played  in  a 
representative  combined  arms  environment  with  the  TES  System 
installed.  The  battlestaff  and  Observer  Controllers  will  use 
these  data  for  conducting  After  Action  Reviews. 
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Basis  for  Measure  The  basis  for  this  measure  is  determined  by 
OPORD  1-95,  Annex  F,  1.  e.,  and  the  HTI-ORD,  Section  1., 
regarding  the  projected  requirements  for  Army  Standardized  After 
Action  Review  products. 

Complimentary  Measure  (CM)  4.1  User  ratings  regarding  the 
ability  of  the  TES  System  to  provide  accurate,  readily 
interpreted  data  to  be  used  in  the  preparation  and  presentation 
of  an  Army  After  Action  Review. 

CM  Methodology  Performance  feedback  capabilities  will  be 
assessed  via  the  use  of  rating  questionnaires  administered  to  the 
aircrews  and  the  battlestaff  representing  the  various  echelons 
supported  by  the  TES  System.  The  battlestaff  members  and 
Observer  Controllers  involved  in  conducting  the  After  Action 
Review  shall  answer  questions  about  how  well  the  TES  System 
supports  their  ability  to  conduct  a  thorough  and  meaningful  AAR. 
The  pilots  will  answer  questions  regarding  how  well  the  TES 
system  presents  the  necessary  information  for  a  thorough  and 
meaningful  AAR.  The  test  control  personnel  will  attend  and 
review  the  AARs  for  completeness.  The  test  control  personnel 
will  compare  the  recorded  real-time  battle  exercise  information 
to  the  data  available  for  the  purpose  of  preparing  and  presenting 
the  After  Action  Review.  The  test  control  personnel  will  render 
opinions  by  the  application  of  military  judgment  to  augment  the 
rating  questionnaire  data  when  necessary.  The  data  will  be 
aggregated  from  questionnaires  and  video/audio  tapes.  The  data 
will  be  reduced  and  stored  in  the  performance  database.  Mean 
values  will  be  calculated  separately  for  the  pilots  and  for  the 
Battlestaff /OCs  and  provided  as  output. 


Additional  Operational  Issue  (AOI)  5 

Can  the  TES  System  be  sustained  in  an  operational  environment? 

Scope  This  issue  assesses  the  sustainability  of  the  TES  System, 
RAM,  and  the  adequacy  of  the  logistics  support  plan.  Data  will 
be  collected  on  the  ten  (10)  TES  System  Aircraft  Instrumentation 
Package  sets  designated  for  the  V&V  effort.  The  logistics 
support  plan  is  for  two  levels  of  maintenance:  unit  and 
contractor  support.  Unit  level  maintenance  will  be  soldier 
supported  and  limited  to  installation,  cleaning,  normal 
operation,  and  simple  troubleshooting.  The  logistics  support 
plan  provides  for  the  use  of  a  Contractor  Logistics  Support  (CLS) 
program  for  major  maintenance  and  logistical  support  for  the  life 
of  the  system.  Since  the  "A-kit"  portion  of  the  TES  System  AIP 
consists  largely  of  software  imbedded  within  organic  aircraft 
components,  maintenance  data  will  be  collected  for  both  unit  and 
contractor  levels  of  maintenance.  Data  will  be  collected  during 
single-ship  gunnery  tables  and  during  company  and  battalion  size 
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maneuver  exercises  played  in  a  representative  combined  arms 
environment . 

Complimentary  Measure  (CM)  5.1  TES  System  Mean  Time  Between 
Mission  Affecting  Failure  (MTBMAF) .  System  evaluation  criteria: 
No  Go,  <  50  hours. 

Basis  for  Measure  The  basis  for  this  measure  is  determined  by 
the  Longbow  TES  System  Requirements  Document ,  Paragraph  3, 
regarding  RAM  requirements . 

CM  Methodology  Data  will  be  collected  and  aggregated  from  RAM 
data  collection  sheets.  Upon  completion  of  each  test  event,  the 
data  will  be  reduced  and  stored  in  the  RAM  database.  The  MTBMAF 
for  each  TES  System  will  be  calculated  and  provided  as  output. 

In  addition,  the  mean  MTBMAF  will  be  calculated  and  provided  as 
output  for  the  ten  (10)  TES  System  AIPs  designated  for  the  test. 
The  MTBMAF  will  be  computed  using  the  following  formula: 


TOT 

MTBMAF  =  - - 

TMAF 


where : 

TOT  =  Total  system  Operating  Time 

TMAF  =  Total  number  of  Mission  Affecting  Failures 

Definitions  of  data  elements: 

TOT  -  Time  during  which  the  TES  System  is  fully  mission 
capable,  and  the  system  or  any  component  is  operating  or  being 
controlled  by  the  operator.  TOT  may  include  time  during  which 
the  TES  System  is  partially  mission  capable  but  is  fully  capable 
of  performing  all  essential  mission  functions.  TOT  does  not. 
include  maintenance  time.  The  TOT  will  be  taken  from  the  TES 
System  RAM  tracking  sheets. 

TMAF  -  The  count  of  MAFs  as  defined  in  the  Failure 
Definitions  and  scored  by  the  test  control  personnel. 

Complimentary  Measure  (CM)  5.2  TES  System  Mean  Time  To  Repair 
(MTTR) .  System  evaluation  criteria:  No  Go,  <  0.5  hours  per 
failure. 

Basis  for  Measure  The  basis  for  this  measure  is  determined  by 
the  Longbow  TES  System  Requirements  Document,  Paragraph  3, 
regarding  RAM  requirements. 


113 


CM  Methodology  The  Mean  Time  To  Repair  (MTTR)  is  generally  used 
to  quantify  a  maintainability  characteristic  for  repairable 
systems.  The  maximum  Mean  Time  to  Repair  is  defined  as  the  time 
below  which  all  corrective  maintenance  tasks  must  be  completed. 

It  is  an  on-system  parameter  and  does  not  include  off-system 
repair  of  replaced  components.  This  measure  is  used  to  compute 
both  unit  and  contractor  level  support  on-system  MTTR  for  the  TES 
System.  System  component  MTTRs  will  be  evaluated  for  their 
relative  impact  on  TES  System  maintainability  and  availability. 
Data  will  be  collected,  aggregated,  and  reduced  similar  to  CM 
5.1.  The  MTTR  will  be  computed  using  the  following  formulas: 


MTTR 


TTR 


(Unit) 


(Unit) 


TCMA 


(Unit) 


MTTR 


TTR 


(Contract) 


(Contract) 


TCMA 


(Contract) 


where : 

TTR  =  Time  To  Repair 

TCMA  =  Total  Corrective  Maintenance  Actions 
MTTR  =  Mean  Time  To  Repair 

Definitions  of  data  elements: 

MTTR  -  The  mean  time  required  to  perform  corrective 
maintenance  actions . 

TTR  -  Total  unit  or  contractor  on-system  corrective 
maintenance  clock  time. 

TCMA  -  Total  number  of  corrective  maintenance  actions 
performed  by  unit ~or  contractor  support. 

Complimentary  Measure  5.3  Operational  availability  (A0)  of  the 
TES  System.  System  evaluation  criteria:  No  Go,  <  98  percent 
(based  on  an  operating  time  of  586  flight  hours  per  year) . 

Basis  for  Measure  The  basis  for  this  measure  is  determined  by  - 
the  Longbow  TES  System  Requirements  Document,  Paragraph  3, 
regarding  RAM  requirements . 

CM  Methodology  This  measure  examines  the  TES  System  demonstrated 
availability  characteristics.  Operational  availability  is 
generally  defined  as  an  objective  determination  of  how  well  a 
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system  meets  specified  performance  requirements.  A  TES  System 
AIP  may  only  be  in  one  availability  category  at  any  time.  Data 
will  be  collected,  aggregated,  and  reduced  similar  to  CM  5.1.  Ao 
will  be  computed  using  the  following  formula: 


Ao 


TOT  +  TST 

TOT  +  TST  +  TCMT  +  TPMT  +  TALDT 


where : 

TOT  =  Total  Operating  Time 

TST  =  Total  Standby  Time 

TCMT  =  Total  Corrective  Maintenance  Time 

TPMT  =  Total  Preventive  maintenance  Time 

TALDT  =  Total  Administrative  and  Logistics  Delay  Time 

Definitions  of  data  elements: 

TOT  -  As  previously  defined  (CM  5.1). 

TST  -  Time  during  which  the  TES  System  is  capable  of 
performing  all  essential  functions  but  is  not  in  TOT  status.  TST 
includes  time  when  the  system  is  capable  of  operation,  but  TST 
does  not  include  maintenance  time. 

TCMT  -  Time  during  which  the  TES  System  is  under  active 
maintenance  to  correct  a  deficiency.  TCMT  does  not  include 
preventive  maintenance  time  or  administrative  and  logistics  delay 
time . 


TPMT  -  Time  during  which  the  TES  System  is  capable  of  full 
operation  but  is  receiving  preventive  maintenance  or  scheduled 
services  as  defined  in  the  appropriate  technical  manual.  TPMT 
does  not  include  corrective  maintenance  time. 

TALDT  -  Time  during  which  the  TES  System  is  not  capable  of 
full  operation  and  is  not  being  actively  maintained.  TALDT 
includes  time  waiting  for  the  maintainer  and  waiting  for  repair 
parts.  The  TALDT  will  be  computed  from  the  administrative  and 
logistics  delay  times. 

Complimentary  Measure  5.4  Probability  of  the  TES  System 
successfully  completing  a  six  hour  training  mission  without  an 
operational  failure  (Ps)  .  System  evaluation  criteria:  No  Go,  < 
89  percent. 

Basis  for  Measure  The  basis  for  this  measure  is  determined  by 
the  Longbow  TES  System  Requirements  Document,  Paragraph  3, 
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regarding  RAM  requirements. 

CM  Methodology  This  measure  assesses  the  reliability  stated  in 
terms  of  a  probability  of  completing  a  six  hour  mission.  The 
expression  for  the  reliability-function  for  the  TES  System  is 
derived  from  the  fact  that  the  system  is  composed  primarily  of 
electrical  ( non-mechanical )  equipment.  The  cumulative  failure 
distribution,  or  simply  the  probability  of  failure  as  a  function 
of  time  for  electronic  equipment  generally  exhibits  the  character 
of  a  negative  exponential  during  its  useful  life.  The  failure 
distribution  F(t)  is  the  probability  that  the  system  will  fail  by 
time  t.  It  is  defined  by  expression: 


F(t)  =  £f  (t)dt 

The  reliability  function  R(t)  is  the  probability  that  the  system 
will  survive  beyond  time  t.  It  is  defined  by  the  following 
expression: 

R (t )  =  1  -  F (t )  or 

R  ( t )  =  j”f  ( t )  dt 

Since  the  time  to  failure  is  described  by  a  negative  exponential 
function,  then 


R  ( t )  =  f  —  e  t/M  dt 
*  M 

Where  M  is  the  mean  life  (mean  time  between  failures)  and  t  is 
the  period  of  interest.  So 


R  (t)  =  e"t/M 

Data  will  be  collected,  aggregated,  and  reduced  similar  to  CM 
5.1.  The  probability  of  completing  a  six  hour  mission,  and  the 
demonstrated  reliability  will  be  computed  from  the  previously 
derived  formula: 


Ps 


e~(6/  MTBMAF) 


Re 


e 


—(HOT  /  MTBMAF) 
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where : 

Re  =  Demonstrated  reliability 
e  =  Natural  logarithm 

MTBMAF  =  Mean  Time  Between  Mission  Affecting  Failures 
TOT  =  Total  Operating  Time 

Complimentary  Measure  (CM)  5.5  TES  System  maintenance  ratio 
(MR).  System  evaluation  criteria:  No  Go,  <  2.7  percent. 

Basis  for  Measure  The  basis  for  this  measure  is  determined  by 
the  Longbow  TES  System  Requirements  Document,  Paragraph  3, 
regarding  RAM  requirements . 

CM  Methodology  The  Maintenance  Ratio  is  a  measure  of  the  total 
maintenance  manpower  burden  required  to  maintain  the  TES  System 
in  an  operational  environment.  The  system  must  display  a  MR  that 
does  not  result  in  a  significant  maintenance  burden  on  the  unit. 
Data  will  be  collected,  aggregated,  and  reduced  similar  to  CM 
5.1.  The  MR  will  be  computed  and  summarized  for  both  unit  and 
contractor  maintenance  according  to  the  following  formulas: 


TMMH  (Unit) 
MR  (Unit)  —  - 

TOT 


MR 


(Contract) 


TMMH  (Contract) 

TOT 


where: 

TMMH  =  Total  Maintenance  Man  Hours 
TOT  =  Total  Operating  Time 

Definitions  of  data  elements: 

TMMH  -  Total  maintenance  man-hours  by  unit  and  contractor, 
includes  both  corrective  and  preventive  maintenance. 

TOT  -  As  previously  defined  (CM  5.1) . 

Complementary  Measure  5.6  Adequacy  of  the  TES  System  support 
equipment  and  the  TMDE. 

Basis  for  Measure  The  basis  for  this  measure  is  determined  by 
the  Longbow  TES  System  Requirements  Document,  Paragraph  3, 
regarding  RAM  requirements . 
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CM  Methodology  This  issue  assesses  the  interim  support  package 
and  TMDE .  The  logistics  support  hardware  and  software  needed  to 
support  the  system  will  be  examined.  Hardware  includes  tools  and 
test  equipment.  Software  includes  technical  manuals,  repair 
parts  and  special  tools  listings.  The  adequacy  of  the  support 
equipment  and  TMDE  will  be  assessed  via  the  use  of  an  informal 
review  checklist.  The  review  checklist  consists  of  questions 
presented  in  a  checklist  format,  designed  to  assess  whether  the 
appropriate  characteristics  have  been  considered  and  incorporated 
into  the  TES  System  support  equipment  and  TMDE.  The  results  will 
be  entered  and  stored  in  the  RAM  database. 


Additional  Operational  Issue  (AOI)  6 

Can  the  TES  System  Aircraft  Instrumentation  Package  B-kit  be 
installed  and  removed  in  an  operational  environment? 

Scope  This  issue  assesses  the  installability  of  the  TES  System. 
Data  will  be  collected  in  an  operational  environment  for  the  time 
required  to  install  the  system  for  operation,  and  to  de-install 
the  system  for  movement. 

Complimentary  Measure  6.1  Time  required  for  installation. 

System  evaluation  criteria:  No  Go,  >  90  minutes. 

Basis  for  Measure  The  basis  for  this  measure  is  determined  by 
the  Longbow  TES  System  Requirements  Document,  Paragraph  2, 
regarding  technical  capabilities. 

CM  Methodology  Installation  will  be  measured  by  the  amount  of 
time  required  for  a  standard  crew  of  three  maintenance  personnel 
to  bring  the  TES  System  B-kit  to  preflight-ready  status  from 
travel  configuration  (in  transit  crates) .  Data  collectors  will 
record  the  time  to  perform  the  installation.  Test  personnel  will 
take  photographs  and  video  tape  recordings  of  crew  actions  during 
the  installation  to  document  the  process.  Additionally,  data 
collectors  will  gather  data  by  the  use  of  an  informal  review 
checklist  designed  to  assess  whether  the  TES  System  design 
features  have  an  impact  on  installation.  The  installation  times 
and  the  review  checklists  will  be  entered  into  the  MANPRINT 
database.  The  mean  values  for  the  installation  time  and 
observations  from  the  review  checklists  will  be  provided  as 
output.  The  installation  time  will  be  calculated  using  the 
following  formula: 


Installation  Time  =  Start  Time  -  Stop  Time 
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Definitions  of  data  elements: 

Start  Time  =  When  the  TES  System  B-kit  is  uncrated 
(removed  from  the  transit  cases) . 

Stop  Time  =  When  the  TES  System  B-kit  is  fully 
installed  without  restrictions  or  limitations  that  preclude  the 
system  from  being  mission  ready. 

Complimentary  Measure  6.2  Time  required  for  removal.  System 
evaluation  criteria:  No  Go,  >  60  minutes. 

Basis  for  Measure  The  basis  for  this  measure  is  determined  by 
the  Longbow  TES  System  Requirements  Document,  Paragraph  2, 
regarding  technical  capabilities. 

CM  Methodology  Same  as  for  CM  6.1  using  a  crew  of  three 
maintenance  personnel.  The  time  required  for  removal. 


Removal  Time  =  Start  Time  -  Stop  Time 


Definitions  of  data  elements: 

Start  Time  =  When  the  crew  commences  to  remove  the  TES 
System  B-kit  from  the  aircraft. 

Stop  Time  =  When  the  TES  System  AIP  has  been  removed  and 
brought  to  travel  configuration  (B-kit  packed  in  the  transit 
cases)  . 


A 


Additional  Operational  Issue  (AOI)  7 

Does  the  TES  System  Central  Collection  Facility  Van  have  the 
required  technical  physical  characteristics? 

Scope  This  issue  assesses  of  the  physical  characteristics  of  the 
TES  System  CCF,  to  include  power,  weight  and  stability,  heating, 
ventilation,  and  air  conditioning  (HVAC) ,  secure  lighting, 
seating  capacity,  and  the  impacts  of  growth  potential  in  these 
areas . 

Complimentary  Measure  (CM)  7.1  The  physical  characteristics  of 
the  TES  System  Central  Collection  Facility. 

Basis  for  Measure  The  basis  for  this  measure  is  determined  by 
the  TES  Master  Plan,  '  Section  5-4,  regarding  the  implications  of 
future  TES  requirements. 
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CM  Methodology  This  measure  examines  the  technical  parameters  of 
the  TES  System  CCF  via  the  use  of  an  informal  review  checklist 
designed  to  assess  whether  the  appropriate  characteristics  and 
Human  Factors  Engineering  have  been  considered  and  incorporated 
into  the  design  of  the  TES  System  CCF.  Additionally,  photographs 
will  be  used  to  augment  the  checklist  data.  Observations  from 
the  review  checklists  will  be  entered  into  the  MANPRINT  database 
and  provided  as  output. 


Additional  Operational  Issue  (AOI)  8 

Is  the  TES  System  suitable  for  operators  and  maintainers? 

Scope  This  issue  compares  the  design  of  the  TES  System  with  the 
capabilities  of  the  target  audience  operators  and  maintainers. 
Application  of  HFE  considers  the  efficiency  of  the  hardware  and 
software  interfaces  with  the  soldier,  the  task  environment,  the 
task  characteristic,  and  how  these  factors  affect  the  workload  of 
the  soldier. 

Complimentary  Measure  8.1  Assessment  of  observed  Human  Factors 
Engineering  (HFE)  deficiencies. 

Basis  for  Measure  The  basis  for  this  measure  is  determined  by 
the  Longbow  TES  System  Requirements  Document,  Paragraph  4, 
regarding  Human  Factors  Engineering  requirements . 

CM  Methodology  HFE  questionnaires  and  interviews  will  be 
administered  to  the  system  operators  and  maintainers  to  obtain 
opinions  regarding  the  human  engineering  design.  Each  TES  System 
operator  (pilot)  and  maintainer  will  complete  the  questionnaire 
at  least  once  during  the  V&V.  Additionally,  information  will  be 
gathered  by  the  use  of  an  informal  checklist  designed  to  assess 
whether  the  appropriate  HFE  engineering  has  been  considered  and 
incorporated  into  the  design  of  the  TES  System.  Photographs  will 
be  used  to  document  any  foreseen  problems  with  the  human  factors 
engineering.  Observations  and  interviews  will  be  entered  into 
the  MANPRINT  database. 

Complimentary  Measure  8.2  Assessment  of  the  physical 
characteristics  of  the  TES  System  B-kit  transit  cases. 

Basis  for  Measure  The  basis  for  this  measure  is  determined  by 
the  Longbow  TES  System  Requirements  Document,  Section  2, 
regarding  the  technical  capabilities. 

CM  Methodology  This  measure  assesses  the  portability  of  the  TES 
System  B-kit  in  transit  cases  (travel  configuration)  which 
satisfy  operational  transportability  two-man  lift  requirements. 
The  V&V  will  provide  observations,  photographs,  and  operator 
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comments  on  the  TES  System  storage  capabilities.  Information 
will  be  gathered  by  the  use  of  an  informal  review  checklist 
designed  to  assess  whether  the  appropriate  HFE  engineering  has 
been  considered  and  incorporated  into  the  design  of  the  TES 
System  transit  cases.  Photographs  will  be  used  to  document  the 
data.  Data  will  be  entered  into  the  MANPRINT  database  and 
provide  as  output. 

Complimentary  Measure  8.3  Assessment  of  observed  manpower 
problems . 

Basis  for  Measure  The  basis  for  this  measure  is  determined  by 
the  Longbow  TES  System  Requirements  Document,  Paragraphs  5  and  6, 
regarding  maintenance  and  manpower. 

CM  Methodology  This  measure  assesses  the  number  of  people  needed 
to  operate,  maintain,  and  support  the  TES  System.  The  assessment 
includes  determining  whether  the  doctrinal  requirements  of  the 
TES  System  are  appropriate  with  respect  to  total  number  of 
soldiers,  MOS,  skill  level,  and  skill  identifiers.  Data  will  be 
collected  by  the  use  of  an  informal  review  checklist,  and  by 
comments,  interviews,  and  observations  as  needed. 

Complementary  Measure  8.4  Assessment  of  observed  personnel 
problems . 

Basis  for  Measure  The  basis  for  this  measure  is  determined  by 
the  Longbow  TES  System  Requirements  Document,  Paragraph  6, 
regarding  manpower. 

CM  Methodology  This  measure  examines  the  experience  of  TES 
System  operators  and  maintainers  as  they  relate  to  observed 
personnel  problems.  Data  will  be  gathered  prior  to  the  start  of 
the  test  to  establish  baselines  for  operator  and  maintainer 
experience  and  proficiency  levels.  These  data  will  be  collected 
on  a  feeder  report  and  stored  in  the  MANPRINT  database. 

Complementary  Measure  8.5  Assessment  of  observed  training 
problems . 

Basis  for  Measure  The  basis  for  this  measure  is  determined  by 
the  Longbow  TES  System  Requirements  Document,  Paragraph  6, 
regarding  manpower  requirements. 

CM  Methodology  ,  This  measure  examines  the  necessary  training  and 
time  required  to  impart  the  knowledge,  skills,  and  abilities  to 
soldiers  for  operation  and  maintenance  of  the  TES  System.  The 
test  control  personnel  will  review  the  New  Equipment  Training 
(NET)  support  package  and  observe  training  conducted  by  the 
system  contractor.  Test  control  personnel  will  examine  initial 
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training  provided  to  operators  and  maintainers,  following 
contractor  conducted  training  and  completion  of  the  V&V. 
Additionally,  the  test  control  personnel  will  gather  information 
using  an  informal  review  checklist  and  by  conducting  interviews 
with  the  operators,  maintainers,  and  instructors  regarding  the 
adequacy  of  training,  training  devices,  training  materials,  and 
the  user  acceptability  of  training  manuals.  Training  adequacy  is 
assessed  in  terms  of  operator  and  maintainer  proficiency  in 
performing  the  tasks  required  to  effectively  employ  the  TES 
System.  Any  training  devices,  training  publications  and 
literature,  and  methods  of  instruction  included  in  the  Program  Of 
Instruction  (POI)  will  be  addressed.  Data  will  be  entered  into 
the  MANPRINT  database  and  provided  as  output. 

Complementary  Measure  8.6  Assessment  of  safety  problem  severity. 

Basis  for  Measure  The  basis  for  this  measure  is  determined  by 
the  Longbow  TES  System  Requirements  Document,  Paragraph  8, 
regarding  safety  requirements. 

CM  Methodology  This  measure  examines  any  system  characteristics 
detailed'  during  the  test  that  could  cause  injury  to  the  operator 
or  maintainer  of  the  TES  System.  The  test  control  personnel  will 
observe  all  V&V  events  and  will  record  data  as  needed  on  a  Test 
Incident  Report  (TIR)  form.  Additionally,  photographs  will  be 
used  to  document  any  data  if  necessary.  Any  unsolved,  serious 
safety  problem  will  be  cause  for  a  test  halt  until  the  problem  is 
corrected. 

Complementary  Measure  8.7  Assessment  of  health  hazards. 

Basis  for  Measure  The  basis  for  this  measure  is  determined  by 
the  Longbow  TES  System  Requirements  Document,  Paragraph  8, 
regarding  safety  requirements. 

CM  Methodology  This  measure  examines  any  condition  detected 
during  the  operation  or  maintenance  of  the  TES  System  during 
testing  that  can  cause  death,  acute  or  chronic  illness, 
disability,  or  reduced  job  performance  of  the  soldier.  The  test 
control  personnel  will  observe  all  V&V  events  and  will  record  any 
situation  or  condition  observed  during  testing  as  needed  on  a 
Test  Incident  Report  (TIR)  form.  Additionally,  Photographs  will 
be  used  to  document  any  data  if  necessary.  Any  unsolved,  serious 
health  condition  will  be  cause  for  a  test  halt  until  the 
condition  is  corrected. 

Complementary  Measure  8.8  Assessment  of  observed  soldier 
survivability  problems. 

CM  Methodology  This  measure  examines  the  survivability  of  the 
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soldier  and  TES  System  while  conducting  training.  The  test 
control  personnel  will  observe  all  V&V  events  and  will  note  any 
problem  areas.  Any  situation  or  condition  observed  during 
testing  or  noted  during  interviews  or  on  questionnaires  that 
could  potentially  reduce  soldier  or  TES  System  survivability  will 
be  documented  on  a  Test  Incident  Report  (TIR)  form. 

Additionally,  Photographs  will  be  used  to  document  any  data  if 
necessary.  These  data  will  be  stored  in  the  MANPRINT  database. 


123 


FAILURE  DEFINITIONS 


General 

Failure.  Definitions  specify  the  parameters  relative  to 
classification  and  assignment  of  all  system  failures.  Failure 
definitions  are  developed  to  ensure  that  all  program  participants 
are  made  aware  of  these  definitions  and  criteria.  It  is  the  key 
to  defining  failures  and  mission  essential  functions  relative  to 
the  RAM  issues  for  OT.  The  mission-essential  function 
descriptions  for  this  test  are  held  to  a  minimum  consistent  with 
the  proposed  set  of  test  procedures.  The  Program  Manager  or 
decision  making  authority  will  be  the  final  approving  authority 
of  the  Failure  Definitions 

Failure  Classification 

Mission  Affecting  Failure  A  Mission  Affecting  Failure  (MAF)  is 
one  which  prevents  the  aircrew  from  performing  any  aspect  of 
weapons  employment  or  weapons  initialization  using  the  proper 
procedures  prescribed  by  the  operator' s  manual  or  any  other 
pertinent  training  publication.  The  loss  of  a  mission-essential 
function  is  one  that  is  readily  identifiable  by  the  aircrew 
during  normal  operations . 

Non-Mission  Affecting  Failure  A  non-Mission  Affecting  Failure  is 
one  which  does  not  prevent  the  aircrew  from  performing  weapons 
employment  or  weapons  initialization  but  requires  them  to  perform 
the  required  procedure (s)  in  a  degraded  mode.  The  test  officer' 
will  establish  the  maximum  allowable  degradation  of  the  mission-  . 
essential  functions  on  a  case  by  case  basis. 


Failure  Categories 

Equipment  Design  Failure  Any  failure  which  can  be  traced 
directly  to  the  design  of  the  equipment;  that  is,  the  design  of 
the  equipment  caused  the  part  in  question  to  degrade  or  fail, 
resulting  in  an  equipment  failure. 

Equipment  Manufacturing  Failure  A  failure  caused  by  poor 
workmanship  or  inadequate  manufacturing  process  control  during 
equipment  construction,  testing,  or  repair  prior  to  the  start  of 
testing. 

Design  Failure  The  failure  of  parts  which  can  be  traced  directly 
to  inadequate  design. 

Part  Manufacturing  Failure  Part  manufacturing  failures  are  the 
result  of  poor  workmanship  or  inadequate  manufacturing  process 
control  during  part  assembly,  inadequate  inspections,  or  improper 
testing. 
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Software  Error  Failure  A  failure  caused  by  an  error  in  the. 
computer  program  associated  with  the  hardware. 

Failure  Response 

Problem  and  Failure  Action  The  occurrence  of  a  problem  or 
failure  that  affects  satisfactory  operation  of  the  equipment 
shall  be  entered  on  a  TIR.  The  failed  equipment  shall  be  removed 
from  the  test  with  minimum  interruption  to  the  equipment 
continuing  on  test. 

Problem  and  Failure  Reporting  A  failure  report  shall  be 
initiated  at  the  occurrence  of  each  problem  or  failure  of 
contractor  hardware  and  software  on  a  TIR.  The  report  shall 
contain  the  information  required  to  permit  determination  of  the 
origin  and  correction  of  failures.  Descriptions' of  failure 
symptoms,  conditions  surrounding  the  failure,  failed  hardware 
identification,  and  operating  time  at  time  of  failure  shall  be 
entered  on  a  TIR.  Additionally,  the  description  should  contain 
all  repair  actions  taken  to  return  the  item  to  operational 
readiness. 

Failure  Verification  Reported  failures  shall  be  verified  as 
actual  failures  or  an  acceptable  explanation  provided  to  the 
Program  Manager  for  lack  of  failure  verification.  Failure 
verification  is  determined  either  by  repeating  the  failure  mode 
on  the  reported  item  or  by  physical  or  electrical  evidence  of 
failure.  Lack  of  failure  verification,  by  itself,  is  not 
sufficient  rationale  to  conclude  the  absence  of  a  failure. 

Corrective  Action  When  the  cause  of  failure  has  been  determined, 
a  corrective  action  shall  be  developed  to  eliminate  or  reduce  the 
recurrence  of  the  failure.  Repairs  shall  be  made  in  accordance 
with  normal  field  operating  procedures  and  manuals.  The  procuring 
activity  shall  review  the  corrective  actions  and  the  status  prior 
to  implementation.  In  all  cases,  the  failure  analysis  and  the 
resulting  corrective  actions  shall  be  documented.  The 
effectiveness  of  the  corrective  action  should  be  demonstrated  by 
restarting  the  test  at  the  beginning  of  the  test  cycle  in  which 
the  original  failure  occurred. 

Corrective  Maintenance  (repair)  The  actions  performed,'  as  a 
result  of  failure  to  restore  an  item  to  a  specified  condition. 

Failure  categories/failure  responses  were  cited  from  MIL-STD 
781D . 
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DATA  MANAGEMENT 


Overview 

The  data  management  process  condenses  and  formats  the  V&V  data 
for  analysis  and  documentation?  A  preliminary  level- four 
database  structure  has  been  developed  to  analyze  and  store  test 
data.  The  test  database  is  organized  in  three  categories: 
performance,  RAM  and  MANPRINT.  Each  data  category  will  be 
collected  and  stored  in  a  separate  database. 


Database  Descriptions 

DBMS  The  DataBase  Management  System  (DBMS)  is  a  central  file 
that  provides  access  to  each  of  the  databases.  DBMS  stores  and 
enables  access  to  questionnaire  formats,  data  collection  forms, 
review  checklists,  and  other  pertinent  forms. 

Performance  The  performance  database  is  used  to  aggregate  TES 
System  performance  data  for  COI  1,  COI  2,  AOI  3,  and  AOI  4.  The 
Performance  database  contains  a  record  for  each  of  30  pilots,  12 
battlestaff  personnel,  and  5  observer  controllers.  The  database 
makes  provisions  to  incorporate  additional  records  as  needed. 

RAM  The  RAM  database  provides  a  consistent  methodology  for 
collecting,  processing,  and  reporting  required  RAM  test  data  for 
AOI  5.  The  RAM  database  contains  a  record  for  each  of  the  10  TES 
System  Aircraft  Instrumentation  Package  sets  designated  for  the 
original  V&V  effort  and  makes  provisions  to  incorporate 
additional  records  as  needed. 

MANPRINT  The  MANPRINT  database  will  be  used  for  collection  and 
processing  data  For  AOI  6,  AOI  7,  and  AOI  8. 


Database  Structure 

The  DBMS  and  the  performance,  RAM,  and  MANPRINT  databases  have 
been  created  in  Microsoft®  Excel  97  and  Microsoft®  Word  97.  The 
database  structure  has  been  constructed  by  creating  Hyperlinks 
(dynamic  date  exchange  and  object  linking  and  embedding)  between 
the  individual  files.  The  databases  are  menu-driven  for  ease  of 
use  and  are  password  protected.  They  are  exportable  to  test 
sites  on  stand-alone  laptop  or  notebook  personal  computers  using 
standard  magnetic  media. 

Data  Reduction 

Performance  Video  data  reduction  consists  of: 

.  VRS  recordings  of  aircraft  symbology,  icons,  messages, 
and  TES  System  operator  conversations. 

.  video  and  audio  recordings  of  CCF  workstation 
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monitors . 

•  video  and  audio  recordings  of  the  actions  and 
conversations  of  the  battlestaff  within  the  CCF. 

.  video  and  audio  recordings  of  the  actions  of  the 
maintenance  personnel. 

Data  identified  by  this  process  is  manually  entered  in  the 
databases.  Questionnaire  data  and  mean  values  will  be  calculated 
and  provided  as  output. 

RAM  Data  reduction  begins  with  an  accurate  transcription  of  the 
data  from  the  data  collection  forms  into  the  RAM  database.  The 
RAM  database  will  generate  all  required  calculations  for  each  TES 
System.  In  addition,  mean  values  will  be  calculated  and  provided 
as  output  for  the  ten  (10)  TES  System  AIPs  designated  for  the 
test . 

MANPRINT  Data  reduction  personnel  will  generate  reports  based  on 
informal  review  checklists. 


COI  Data  Presentation 

The  results  of  the  qualitative  criteria  for  each  Critical 
Operational  Issue  will  be  presented  on  one  chart.  The  criteria 
will  not  be  averaged  to  determine  a  quantitative  value  for  the 
overall  COI.  The  results  of  the  evaluation  of  the  Critical 
Operational  Issues  will  be  used  for  the  accreditation  process, 
consequently,  the  decision  maker  must  place  a  relative  weight  on 
each  supporting  criterion.  The  visual  presentation  of  the 
criteria  scores  for  the  COIs  will: 

.  prompt  the  Longbow  TES  System  proponent  to  carry  out  a 
causality  analysis  if  a  criterion  receives  a  poor  rating. 

•  assist  in  steering  the  terms  of  a  conditional 
accreditation  if  necessary. 

.  provide  a  starting  point  for  the  next  iteration  of  the 
TES  System  development  process  if  required. 


Data  Storage 

Upon  completion  of  the  V&V  effort,  data  will  be  archived  and 
printed  for  submission  to  the  Program  Manager  for  subsequent 
processing  and  analysis. 


Data  Authentication 

The  data  authentication  team  will  consist  of  independent  experts 
with  a  broad  spectrum  of  technical  disciplines.  This  team  will 
assemble  for  the  purpose  of  assessing  and  monitoring  data 
reduction,  quality  control,  and  the  identification  and  analysis 
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of  anomalies  in  the  system,  instrumentation,  database  structures, 
and  test  data. 
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APPENDIX  C 


DATA  COLLECTION  FORMS 


Table  C-l 

Data  Collection  requirements 


Data 

Collection 

Form 

Completed  By 

When  Completed 

COI/AOI 

. 

1 

Aircrew 

Post  Mission 

COI  2,  AO I  4  I 

2 

Battles taf f /OCs 

During/Post 

Mission 

AO I  3,  AO I  4  | 

3 

Participating 

Test  Personnel 

Prior  to  Test 
Events 

Supporting  Test, 

AO  I  8 

4 

Data  Collection 
Personnel 

Continuously 
During  Test 

AO  I  5 

5 

Data  Collection 

Personnel/Test 

Particioants 

Continuously 
During  Test 

Supporting  Test, 

AO  I  8 

6 

Data  Collection 
Personnel 

During  Test 

AO  I  7 

7,8 

Data  Collection 
Personnel 

During  Test 

AO  I  5,6 

9 

Data  Collection 
Personnel 

During  Test 

AO  I  5 

10 

Data  Collection 
Personnel 

During  Test 

COI  2 

g— | 

Data  Collection 
Personnel 

During  Test 

AO  I  8 

15 

Data  Collection 
Personnel 

During  Test 

Environmental 
Impact;  Does  Not 
Support  COI/AOI 
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DATA  COLLECTION  FORM  1 


{PILOT  QUESTIONNAIRE,  PAGE  1) 


TES  System  Pilot  Survey  Questionnaire 

Respondent  Control  Number: _ 

Statements 


1. 


2. 


3. 


4. 


5. 

6. 

7. 

8. 

9. 

10. 

11. 

12. 


Weapons  initialization  procedures  were  the  same 
as  normal  operations  with  the  TES  System 
installed. 

□ 

□ 

□ 

□ 

□ 

□ 

Weapons  selection  procedures  were  the  same  as 
normal  operations  with  the  TES  System 
installed. 

□ 

□ 

□ 

□ 

□ 

□ 

Weapons  symbology  presented  was  the  same  as 
normal  operations  with  the  TES  System 
installed. 

□ 

□ 

□ 

□ 

□ 

□ 

Weapons  status  information  presented  was  the 
same  as  normal  operations  with  the  TES  System 
installed. 

□ 

□ 

□ 

□ 

□ 

□ 

Weapons  range  was  the  same  as  normal 
operation  with  the  TES  System  installed. 

□ 

□ 

□ 

□ 

□ 

□ 

Weapons  range  in  adverse  weather  conditions 
was  the  same  as  normal  operation. 

□ 

□ 

□ 

□ 

□ 

□ 

Simulation  of  weapons  flash  was  the  same  as 
actual  weapons  flash. 

□ 

□ 

□ 

□ 

□ 

□ 

Simulation  of  weapons  smoke  was  the  same  as 
actual  weapons  smoke. 

□ 

□ 

□ 

□ 

□ 

□ 

Simulation  of  weapons  noise  was  the  same  as 
actual  weapons  noise. 

□ 

□ 

□ 

□ 

n 

□ 

System  sensors  with  the  TES  System  installed 
responded  the  same  as  normal  operation. 

□ 

□ 

□ 

□ 

□ 

□ 

The  weapons  system  with  the  TES  System 
installed  performed  as  in  normal  operation. 

□ 

□ 

□ 

□ 

□ 

□ 

The  cockpit  environment  with  the  TES  System 
installed  was  the  same  as  in  normal  operation. 

□ 

□ 

□ 

□ 

□ 

□ 
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DATA  COLLECTION  FORM  1 
(PILOT  QUESTIONNAIRE,  PAGE  2) 


13.  In-flight  activities  were  NOT  affected  by  the 

TES  System  equipment. 

□ 

□ 

□ 

□ 

□ 

□ 

14.  For  a  simulation  system,  TES  System  required 
realistic  cockpit  activity  during  simulation 
exercises. 

□ 

□ 

□ 

□ 

□ 

□ 

15.  Pilot  workload,  with  TES  System  installed,  was 
nominal. 

□ 

□ 

□ 

□ 

□ 

□ 

16.  Post-flight  checklists  required  normal 
operational  times  with  the  TES  System 
equipment  installed. 

□ 

□ 

□ 

□ 

□ 

□ 

17.  Video  presentations  of  TES  System  data 

displayed  during  the  debriefing,  or  After  Action 

Report  (AAR),  were  easily  interpreted. 

□ 

□ 

□ 

□ 

□ 

□ 

18.  Personnel  or  material  did  not  obstruct  video 
presentations  of  TES  System  data  displayed 
during  the  AAR. 

□ 

□ 

□ 

□ 

□ 

□ 

19.  Video  presentations  of  TES  System  data 

displayed  during  the  AAR  were  accurate  to  the 
degree  necessary  for  training  purposes. 

□ 

□ 

□ 

□ 

□ 

□ 

20.  Video  presentations  of  TES  System  data 

displayed  during  the  AAR  were  easily  prepared 
in  the  given  timeframe  of  the  exercise. 

□ 

□ 

□ 

□ 

□ 

□ 

21.  Video  presentations  of  TES  System  data 

displayed  during  the  AAR  were  effective  for  the 
established  training  goals  and  objectives. 

□ 

□ 

□ 

□ 

□ 

□ 

22.  Noise  levels  in  the  TES  System  CCF  during  the 

AAR  did  not  cause  undue  distraction. 

□ 

□ 

□ 

□ 

□ 

□ 

23.  Noise  levels  in  the  TES  System  CCF  during  the 

AAR  did  not  adversely  affect  hearina  the 
debriefing. 

□ 

□ 

□ 

□ 

□ 

□ 

24.  Temperature  levels  in  the  TES  System  CCF 
during  the  AAR  were  nominal. 

□ 

□ 

□ 

□ 

□ 

□ 
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DATA  COLLECTION  FORM  1 
(PILOT  QUESTIONNAIRE,  PAGE  3) 


25.  Adequate  equipment  for  in-field  presentations  of 

TES  System  data  were  available. 

□ 

□ 

□ 

□ 

□ 

□ 

26.  Negative  habit  transfer  was  not  incurred  from 
training  with  TES  System. 

□ 

□ 

□ 

□ 

□ 

□ 

27.  Artificial  Dilot  requirements  were  not  incurred 
from  training  with  TES  System. 

□ 

□ 

□ 

□ 

□ 

□ 

28.  TES  Svstem  did  not  degrade  the  ODerational 
characteristics  of  the  host  system. 

□ 

□ 

□ 

□ 

□ 

□ 

29.  TES  Svstem  did  not  interfere  with  normal  crew 
duties. 

□ 

□ 

□ 

□ 

□ 

□ 

30.  TES  System  required  crewmembers  to  perform 
the  same  functions  in  an  engagement  sequence 
(to  include  aircraft  orientation)  as  when  engaging 
a  target  with  service  ammunition. 

□ 

□ 

□ 

□ 

□ 

□ 

3 1 .  TES  System-equipped  targets  depicted  the 

appropriate  weapons  effects  when  fired  upon  by 
a  TES  System  weapon  device. 

□ 

□ 

□ 

□ 

□ 

□ 

32.  TES  Svstem  ODeration  did  not  cause  or  lead  to 
premature  failures  of  host  components  or 
systems. 

□ 

□ 

□ 

□ 

□ 

□ 

33.  TES  Svstem  operation  did  not  cause  or  lead  to 
damage  of  host  components  or  systems. 

□ 

□ 

□ 

□ 

□ 

□ 
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DATA  COLLECTION  FORM  2 


(STAFF  QUESTIONNAIRE,  PAGE  1) 
TES  System  Battle  Staff  Survey  Questionnaire 

Respondent  Control  Number: _ 

Statements 


1 .  Adequate  training  was  provided  for  the  operation 
of  the  TES  System  CCF  systems. 

2.  Adequate  time  was  available  for  TES  System 
CCF  system  setup  without  undue  delay  in  the 
simulation  timetable. 

3.  The  TES  System  work  environment  during  a 
simulation  was  free  from  environmental 
distractions  that  were  unrelated  to  the  simulation 
exercise. 

4.  Ventilation  in  the  TES  System  CCF  work 
environment  was  adequate  for  the  equipment 
during  the  simulation  exercise. 

5.  Noise  levels  in  the  TES  System  CCF  work 
environment  were  nominal  for  a  simulation 
exercise. 

6.  Temperature  levels  in  the  TES  System  CCF 
work  environment  were  nominal  for  a  simulation 
exercise. 

7.  The  TES  System  CCF  workstation  design  was 
comfortable  for  the  duration  of  the  exercise. 

8.  Adequate  system  features  were  available  to  meet 
command  and  control  requests  for  specific  data. 

9.  Specific  data  requested  was  available  in  a 
reasonable  timeframe. 

10.  Adequate  system  features  were  available  to 
retrieve  stored  data  in  a  reasonable  timeframe. 

11.  Adequate  system  features  were  available  to 
display  and  distribute  retrieved  data. 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 
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DATA  COLLECTION  FORM  2 


(STAFF  QUESTIONNAIRE,  PAGE  2) 

12.  Display  of  data  during  the  simulation  exercise" 
was  adequate  to  discern  easily  between  exercise 
opponents. 

13.  Display  of  data  during  the  simulation  exercise 
was  adequate  to  discern  easily  between  players 
on  the  same  force. 

14.  Data  display  latency  did  not  adversely  affect 

command  decisions  during  the  exercise. 

15.  Data  display  latency  did  not  reduce  the  realism 
of  the  exercise  when  compared  to  other 
simulation  techniques. 

16.  The  data  displayed  during  the  simulation 
exercise  provided  an  adequate  amount  of 
information  regarding  the  status  of  the 
engagement. 

17.  The  data  displayed  during  the  simulation 
exercise  provided  an  adequate  amount  of 
information  regarding  the  status  of  each  player  in 
the  engagement. 

18.  Video  presentations  of  TES  System  data 
displayed  during  the  debriefing,  or  After  Action 
Report  (AAR),  were  easily  interpreted. 

19.  Video  presentations  of  TES  System  data 
displayed  during  the  AAR  were  not  obstructed 
by  personnel  or  material. 

20.  Video  presentations  of  TES  System  data 
displayed  during  the  AAR  were  accurate  to  the 
degree  necessary  for  training  purposes. 

21.  Video  presentations  of  TES  System  data 
displayed  during  the  AAR  were  easily  prepared 
in  the  given  timeframe  of  the  exercise. 

22.  Video  presentations  of  TES  System  data 
displayed  during  the  AAR  were  effective  for  the 
established  training  goals  and  objectives. 

23.  Noise  levels  in  the  TES  System  CCF  during  the 
AAR  did  not  cause  undue  distraction. 
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□  □□□□□ 

□  □□□□□ 

□  □□□□□ 
□  □□□□□ 

□  □□□□□ 
□  □□□□□ 

□  □  □  □  □  □ 

□  □□□□□ 
□  □□□□□ 

D  D  D  D  D  D 

□  □□□□□ 

□  □□□□□ 


DATA  COLLECTION  FORM  2 


(STAFF  QUESTIONNAIRE,  PAGE  3) 


24.  Noise  levels  in  the  TES  System  CCF  during  the 
AAR  did  not  adversely  affect  hearing  the 
debriefing. 

25.  Temperature  levels  in  the  TES  System  CCF 
during  the  AAR  were  nominal. 

26.  Adequate  equipment  for  in-field  presentations  of 
TES  System  data  were  available. 

27.  Adequate  training  was  provided  for  use  of  in¬ 
field  presentation  equipment. 

28.  Documentation  was  provided  for  troubleshooting 
in-field  presentations  of  TES  System  data. 

29.  Documentation  provided  timely  solutions  for 
troubleshooting  in-field  presentation  problems. 

30.  In-field  presentations  of  TES  System  data  were 
easily  interpreted. 

3 1 .  MILES-equipped  targets  depicted  the 
appropriate  weapons  effects  when  fired  upon  by 
a  TES  System  weapon  device. 

32.  The  unit  was  able  to  operate  in  a  sustained 
simulated  wartime  environment  without 
degradation  while  using  TES  System  and 
employing  unit  equipment  defined  in  the  current 
MTOE. 

33.  The  TES  System  equipment  was  safe  to  operate 
in  an  operational  environment. 

34.  The  TES  System  equipment  was  safe  to  maintain 
in  an  operational  environment. 

35.  TES  System  properly  provided  a  target  kill 
indication. 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

36.  TES  System  provided  adequate  support  to  the 
commander  and  controller  personnel  during  the 
unit’s  participation  at  the  CTC. 


□  □□□□□ 
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DATA  COLLECTION  FORM  3 
(PERSONNEL  DATA  SHEET) 


Feeder  Report 

Tactical  Engagement  Simulation  Syatem 


Page  1  of  1 


Form  Date:  Nov  98 
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DATA  COLLECTION  FORM  4 


(RAM  DATA  COLLECTION  SHEET/EQUIPMENT  STATUS,  PAGE  1) 


RAM  Data  Collection  Sheet 
TESS  Equipment  Status 


Data  Collectors  Name: 


Data  Collectors  PIN: 


4 - U 


System  (AIP  No . )  : 


D  D 

M  M  M 

1 _ 1 _ 1  Date: 

Ml 

wm 

|M 

Test  Phase:  f  J 

_  Equipment  Status 

Task  Status 

■ 

Task  Status 

(for  Active  Maintenance) 

(for  Maintenance  Delay) 

1  =  Operating 

2  *  Standby 

3  =  Active  Maintenance 

4  =  Maintenance  Delay 

5  =»  Non-Charge  able 

Down  Time 

UM/CM 

10/11  »  Diagnosis 

12/13  «  Remove/Replace 

14/15  =  Repair 

16/17  ®  Test/Checkout 

18/19  =*  Scheduled 

Maintenance 

DM/CM 

20/21  m  Logistics  Delay 
22/23  =  Admin  Delay 

24/25  =  Travel  Time 

26/27  =  Deferred  Maint 

Time 

Failure  Class 


1  *  Equipment  Design  Failure 

1  «  Mission  Affecting 

2  ■=  Equipment  Manufacturing  Failure 

2  =  Non-Mission  Affecting 

3  =  Part  Design  Failure 

4  *  Part  Manufacturing  Failure 

5  =*  Software  Error  Failure 

Start  Time 
H  H  M  M 


Stop  Time 
H  H  M  M 


Equip 

Status 


Task 

Status 


Failure  Failure 
Class  Category 


4 - U 


Page  1  of  2 


Form  Date:  Nov  98 
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DATA  COLLECTION  FORM  4 


(RAM  DATA  COLLECTION  SHEET/EQUIPMENT  STATUS,  PAGE  2) 


TESS  Equipment  Status 
RAM  Data  Collection  Sheet 


Page  2  of  2 


Form  Date:  Nov  98 
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DATA  COLLECTION  FORM  5 
(TEST  INCIDENT  REPORT,  PAGE  1) 


Tost  Incident  Report 

Tactical  Engagement  Simulation  System  (TESS) 


Effect  On  Mission: 


BIT  Fault  Detection  Results: 


DATA  COLLECTION  FORM  5 


(TEST  INCIDENT  REPORT ,  PAGE  2) 


Test  Incident  Report 

Tactical  Engagement  Simulation  System  (TESS) 


-  Incident  Description:  - 

Include  the  Following: 

Location  of  Incident. 

Activity  and  flight  mode  at  time  of  incident. 

Environmental  Conditions. 

Description  of  any  prior  incidents  which  may  have  an  effect  on  this  incident. 
Description  of  ervent  and  any  immediate  reactions  and/or  diagnosis. 


DATA  COLLECTION  FORM  5 
(TEST  INCIDENT  REPORT,  PAGE  3) 


Test  Incident  Report 

Tactical  Engagement  Simulation  System  (TESS) 


Incident  Description  (continued) 


DATA  COLLECTION  FOFM  6 
(CCP  REVIEW  CHECKLIST) 


TES  SYSTEM  CENTRAL  COLLECTION  FACILITY 
Design  Characteristics 

Does  the  design  of  the  CCF  consider  the  following:  temperature, 
humidity,  vibration,  shock,  pressure,  wind,  sand,  and  dust?  Have 
the  ranges  and  extreme  conditions  been  specified  and  properly 
addressed  in  design?  Have  the  proper  environmental  profiles  been 
addressed? 

Have  provisions  been  made  to  specify  and  control  noise, 
illumination,  temperature,  and  humidity  in  areas  within  the  CCF 
where  battlestaff  personnel  are  required  to  perform  operational 
tasks? 

Human  Factors  Engineering 

Are  operator  panels  optimally  positioned?  For  personnel  in  the 
standing  position,  panels  and  CRTs  should  be  located  between  40 
and  70  inches  above  the  floor.  Critical  or  precise  controls 
should  be  between  48  and  64  inches  above  the  floor.  For 
personnel  in  the  sitting  position,  panels  should  be  located  30 
inches  above  the  floor. 

Are  equipment  racks  mounted  on  roll-out  slides? 
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DATA  COLLECTION  FORM  7 


(MAINTENANCE  REVIEW  CHECKLIST) 


LONGBOW  TES  SYSTEM  MAINTENANCE  CONCEPT 

Have  the  levels  of  maintenance  been  identified  and  defined? 

Have  basic  maintenance  functions  been  identified  for  each  level? 

Have  level-of-repair  policies  been  established?  Repair  versus 
discard?  Repair  at  unit  level  or  at  contractor  level? 

Have  the  criteria  for  level-of-repair  decisions  been  adequately 
defined? 

Have  the  test  and  support  equipment  requirements  been  defined  for 
each  level  of  maintenance? 

Has  a  detailed  maintenance  task  analysis  been  done  to  verify 
maintenance  task  sequences,  task  complexities  and  personnel 
skills? 

Is  the  detailed  maintenance  task  analysis  compatible  with 
maintainability  data  and  the  logistic  support  plan? 

Are  the  detailed  maintenance  tasks  compatible  with  TES  system 
maintenance  procedures  (task  sequences,  depth  of  explanatory 
material  based  on  task  complexity) ? 

Have  all  system  software  requirements  for  maintenance  functions 
been  identified?  Have  these  requirements  been  developed  through 
a  system-level  functional  analysis  to  provide  traceablity? 
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DATA  COLLECTION  FORM  8 


(DESIGN  REVIEW  CHECKLIST,  PAGE  1) 


TES  SYSTEM  DESIGN  FEATURES 
Factors  impacting  Maintenance 

Accessibility  Are  key  system  components  directly  accessible  for 
the  performance  of  maintenance  tasks? 

Is  access  easily  attained? 

Are  access  requirements  compatible  with  the  frequency  of 
maintenance  or  the  importance  of  the  maintenance  tasks? 

Are  access  doors  provided  where  appropriate?  Are  hinged  doors 
used?  Can  access  doors  that  are  hinged  be  supported  in  the  open 
position? 

Are  access  openings  adequate  in  size  and  optimally  located  for 
the  required  access? 

Are  access  door  fasteners  of  the  quick-release  variety? 

Can  access  be  attained  without  the  use  of  tools? 

If  tools  are  required  to  gain  access,  are  the  number  of  tools 
held  to  a  minimum?  Are  the  tools  standard  or  special? 

Are  access  provisions  between  modules  and  components  adequate? 

Adjustments  and  Alignments  Have  adjustment  and  alignment 
requirements  been  minimized,  or  eliminated? 

Are  adjustment  requirements  known? 

Are  adjustment  points  accessible? 

Are  adjustment -point  locations  compatible  with  the  maintenance 
level  at  which  the  adjustment  is  made? 

Have  adjustment  and  alignment  interaction  effects  been 
eliminated? 

Are  factory  adjustments  specified? 
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DATA  COLLECTION  FORM  8 


(DESIGN  REVIEW  CHECKLIST,  PAGE  2) 


Are  adjustment  points  adequately  labeled?  Can  adjustments  and 
alignments  be  made  without  the  requirement , for  special  tools? 

Calibration  Have  calibration  requirements  been  minimized? 

Are  calibration  requirements  known  where  applicable? 

Are  calibration  frequencies  and  tolerances  known? 

Have  the  facilities  for  calibration  been  identified? 

Are  the  necessary  standards  available  for  calibration? 

Are  calibration  requirements  compatible  with  the  maintenance 
concept  and  the  logistic  support  plan? 

Cables  and  Connectors  Are  cables  fabricated  in  removable 
sections? 

Are  cables  routed  to  avoid  sharp  bends? 

Are  cables  routed  to  avoid  pinching? 

Is  cable  labeling  adequate? 

Is  cable  clamping  adequate? 

Are  the  connectors  quick-disconnect? 

Are  connectors  that  are  mounted  on  surfaces  far  enough  apart  so 
that  they  can  be  firmly  grasped  for  connecting  and  disconnecting? . 

Are  connectors  and  receptacles  labeled? 

Are  connectors  standardized? 

Mounting  Is  it  possible  to  limit  maintenance  to  the  removal  of 
only  the  failed  part  when  a  failure  occurs? 

Is  the  design  compatible  with  level  of  repair  analysis  decisions? 
Repairable  items  are  designed  to  include  maintenance  provisions 
such  as  test  points,  accessibility,  and  plug-in  components. 
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DATA  COLLECTION  FORM  8 


(DESIGN  REVIEW  CHECKLIST,  PAGE  3) 


Are  plug-in  modules  and  components  used  to  the  maximum  extent 
possible? 

Are  accesses  between  modules  adequate  to  allow  for  hand  grasping? 

Are  modules  and  components  mounted  such  that  the  removal  of  any 
single  item  for  maintenance  will  not  require  the  removal  of  other 
items? 

In  areas  where  module  stacking  is  necessary  because  of  limited 
space,  are  the  modules  mounted  in  such  a  way  that  access  priority 
has  been  assigned  in  accordance  with  the  predicted  removal  and 
replacement  frequency?  Items  that  require  frequent  maintenance 
should  be  more  accessible. 

Are  modules  and  components  (not  plug-in)  mounted  with  four  fas¬ 
teners  or  less?  Modules  should  be  securely  mounted,  however  the 
number  of  fasteners  should  be  held  to  a  minimum. 

Are  shock-mounting  provisions  incorporated  where  shock  and 
vibration  requirements  are  excessive? 

Are  provisions  incorporated  to  preclude  installation  of  the  wrong 
module? 

Are  plug-in  modules  and  components  removable  without  the  use  of 
tools?  Are  guides  (slides  or  pins)  provided  to  facilitate  module 
installation? 

Are  modules  and  components  labeled? 

Are  module  and  component  labels  located  on  top  or  immediately 
adjacent  to  the  item  and  in  plain  sight? 

Are  the  labels  permanently  affixed  so  that  they  will  not  come  off 
during  a  maintenance  action  or  as  a  result  of  environment? 

Is  the  information  on  the  label  adequate? 
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DATA  COLLECTION  FORM  8 
(DESIGN  REVIEW  CHECKLIST,  PAGE  4) 


Maintainability  Is  the  system  or  product  maintainable  in  terms 
of  troubleshooting  and  diagnostic  provisions,  accessibility,  ease 
of  replacement  and  handling  capabilities  in  the  performance  of 
maintenance  (corrective  and  preventive)? 

Have  maintainability  requirements  for  the  system  or  equipment 
been  adequately  defined?  Are  they  compatible  with  system 
performance,  reliability,  supportability,  and  effectiveness 
factors? 
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DATA  COLLECTION  FORM  9 


(TMDE  REVIEW  CHECKLIST,  PAGE  1) 


TEST,  MAINTENANCE  AND  DIAGNOSTIC  EQUIPMENT 
Equipment 

Have  standard  TMDE  and  support  equipment  items  been  selected? 

Have  criteria  been  established  for  TMDE  and  support  equipment  at 
each  level  of  maintenance?  Built-in  versus  external  test 
equipment?  Diagnostic  requirements? 


Are  the  selected  TMDE  and  support  equipment  items  compatible  with 
the  prime  equipment?  Does  the  TMDE  do  the  job? 

Are  TMDE  requirements  compatible  with  maintenance  concept, 
logistic  support  plan,  and  level  of  repair  analysis  data? 

Have  TMDE  and  support  equipment  requirements  (both  in  terms  of 
variety  and  quantity)  been  minimized  to  the  greatest  extent 
possible? 

Does  the  system  specification  include  operational  requirements, 
the  maintenance  concept,  and  a  functional  definition  of  the  TMDE? 

Does  the  system  specification  include  effectiveness  requirements 
(reliability,  maintainability,  human  factors,  supportability)  for 
the  TMDE? 

Are  the  reliability  and  maintainability  features  in  the  TMDE 
and  support  equipment  compatible  with  those  equivalent  features 
of  the  TES  System  and  the  prime  equipment? 

Have  logistic  support  requirements  for  the  selected  TMDE  been 
defined?  This  includes  maintenance  tasks,  calibration  equipment, 
spare  and  repair  parts,  personnel  and  training,  data,  and 
facilities? 
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DATA  COLLECTION  FORM  9 


(TMDE  REVIEW  CHECKLIST,  PAGE  2) 


Testability  using  TMDE 

Have  self-test  provisions  been  incorporated  where  appropriate? 

Is  reliability  degradation  due  to  the  incorporation  of  built-in 
test  minimized? 

Is  the  extent  or  depth  of  self-testing  compatible  with  the  level 
of  repair  analysis? 

Are  self-test  provisions  automatic? 

Have  direct  fault  indicators  been  provided  (a  fault  light,  an 
audio  signal,  or  a  means  of  determining  that  a  malfunction 
positively  exists)?  Are  continuous  monitoring  provisions 
incorporated  where  appropriate? 

Are  test  points  provided  to  enable  checkout  and  fault  isolation 
beyond  the  level  of  self-test? 

Are  test  points  accessible? 

Are  test  points  functionally  and  conveniently  grouped  to  allow 
for  sequential  testing  (following  a  signal  flow),  testing  of 
similar  functions,  or  frequency  of  use  when  access  is  limited? 

Are  test  points  provided  for  a  direct  test  of  all  replaceable 
items? 

Are  test  points  adequately  labeled?  Each  test  point  should  be 
identified  with  a  unique  number,  and  the  proper  signal  or 
expected  measured  output  should  be  specified  on  a  label  located 
adjacent  to  the  test  point. 

Can  the  component  malfunctions  that  could  possibly  occur  be 
detected  through  a  no-go  indication  at  the  system  level? 

Will  the  prescribed  maintenance  software  provide  adequate 
diagnostic  information? 
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DATA  COLLECTION  FORM  10 


(HFE  REVIEW  CHECKLIST,  Page  3) 


HUMAN  FACTORS  (TES  SYSTEM  AIRCRAFT  INSTRUMENTATION  PACKAGE) 

Cockpit  Panel  Displays  and  Controls 

Are  controls  or  circuit  breakers  standardized? 

Are  controls  or  circuit  breakers  sequentially  positioned? 

Is  control  spacing  adequate? 

Is  control  or  circuit  breaker  labeling  adequate? 

Have  the  proper  control  and  display  relationships  been 
incorporated,  based  on  sound  human  factors  criteria? 

Are  the  proper  type  of  panel  switches  or  circuit  breakers  used? 

Is  the  control  panel  lighting  adequate? 

Are  the  controls  placed  according  to  frequency  and/or  criticality 
of  use? 

Has  a  system  analysis  been  done  to  verify  optimum  human-machine 
interfaces?  Are  automated  and  manual  functions  adequately 
identified? 

Are  the  identified  automated  and  manual  functions  consistent  with 
the  results  of  the  overall  system-level  functional  analysis? 

Has  a  detailed  operator  task  analysis  been  done  to  verify  task 
sequences,  to  include  pre-flight? 

Are  the  detailed  operator  tasks  compatible  with  the  TES  system 
operating  procedures  (task  sequences,  depth  of  explanatory 
material  based  on  task  complexity) ? 

For  human-interface  functions,  is  the  system  design  optimum  when 
considering  human  sensory  factors,  psychological  factors,  and 
physiological  factors?  For  manual  tasks,  does  the  design  reflect 
ease  of  operation  by  trained  pilots?  Is  the  design  such  that 
potential  human  error  rates  are  minimized  during  operation? 

Is  the  Human  Factors  Engineering  (HFE)  compatible  with  Army 
safety  engineering  requirements? 
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DATA  COLLECTION  FORM  11 


(TRANSPORTABILITY  REVIEW  CHECKLIST) 


HANDLING 

General 

For  heavy  items,  are  hoist  lugs  or  base-lifting  provisions  for 
forklift-truck  application  incorporated?  Hoist  lugs  should  be 
provided  on  all  items  weighing  more  than  150  pounds. 

Are  hoist  and  base-lifting  points  identified  relative  to  lifting 
capacity? 

Are  weight  labels  provided? 


Two  Man  Lift  Requirements 

Are  items  weighing  more  than  40  pounds  provided  with  two  handles 
for  two-man  carrying? 

Are  units,  components,  or  other  items  weighing  over  10  pounds 
provided  with  handles?  Are  the  proper-sized  handles  used  and  are 
they  located  in  the  correct  position?  Are  the  handles  optimally 
located  from  the  weight  distribution  standpoint?  Carrying 
handles  should  be  located  over  the  center  of  gravity. 


Packing 

Do  the  TES  System  transit  crates  protect  vulnerable  components 
from  damage  during  handling? 
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DATA  COLLECTION  FORM  12 


(MANPOWER ,  PERSONNEL,  AND  TRAINING  REVIEW  CHECKLIST) 


MANPOWER/ PERSONNEL/TRAINING 

Have  maintenance  personnel  requirements  (MOS,  quantity  and  skill 
levels)  been  defined? 

Are  operational  and  maintenance  personnel  requirements  minimized 
to  the  greatest  extent  possible? 

Are  operational  and  maintenance  personnel  requirements  compatible 
with  the  logistic  support  plan  and  with  human  factors  data? 

Are  the  planned  personnel  skill  levels  at  each  location 
compatible  with  the  complexity  of  the  operational  and  maintenance 
tasks  specified? 

Has  maximum  consideration  been  given  to  the  use  of  existing 
personnel  skills  for  the  TES  System? 

Have  personnel  effectiveness  factors  been  determined  (actual  time 
that  work  is  accomplished  per  the  total  time  allowed  for  work 
accomplishment) ? 

Have  maintenance  training  requirements  been  specified?  This 
includes  consideration  of  both  initial  training  and  replenishment 
training  throughout  the  projected  TES  life  cycle. 

Have  specific  training  programs  been  planned?  The  type  of 
training,  frequency  of  training,  and  duration  of  training  should 
be  identified. 

Are  the  planned  training  programs  compatible  with  the  personnel 
skill  level  requirements  specified  for  the  performance  of 
operational  and  maintenance  tasks? 

Have  training  equipment  requirements  been  defined  and 
acquisitioned? 

Have  maintenance  provisions  for  training  equipment  been  planned? 
Have  training  data  requirements  been  met? 

Are  the  planned  operating  and  maintenance  procedures  (designated 
for  support  of  the  TES  System  throughout  its  projected  life 
cycle)  used  to  the  maximum  extent  in  the  training  programs? 


152 


DATA  COLLECTION  FORM  13 


(NET  REVIEW  CHECKLIST) 


NEW  EQUIPMENT  TRAINING  (NET)  SUPPORT  PACKAGE 

Has  a  New  Equipment  Training  (NET)  support  Package  been 
developed? 

Have  the  major  elements  of  support  been  defined  (test  and  support 
equipment,  training  devices,  personnel  training,  publications, 
and  technical  data  requirements)? 

Do  the  elements  of  the  NET  package  reinforce  the  system 
maintenance  concept? 

Has  a  detailed  training  plan  for  operator  and  maintenance 
personnel  been  prepared?  Have  training  facility,  equipment, 
material,  software,  and  data  requirements  been  identified? 
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DATA  COLLECTION  FORM  14 


(SAFETY  REVIEW  CHECKLIST) 


SAFETY 

Has  an  integrated  safety  plan  been  prepared  and  implemented? 

Has  a  hazard  analysis  been  accomplished  to  identify  potential 
hazardous  conditions? 

Have  fail-safe  provisions  been  incorporated  in  the  design? 

Have  protruding  devices  been  eliminated  or  are  they  suitably 
protected? 

Have  provisions  been  incorporated  for  protection  against  stray 
voltages?  Are  all  external  metal  parts  adequately  grounded? 

Are  sharp  metal  edges,  access  openings,  and  corners  protected 
with  rubber,  fillets,  fiber,  or  plastic  coating? 

Are  electrical  circuit  interlocks  or  circuit  breakers  employed? 

Are  the  potential  operating  environments  such  that  personnel 
safety  can  be  ensured?  Can  Army  safety  requirements  be 
maintained? 
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DATA  COLLECTION  FORM  15 
(ENVIRONMENTAL  REVIEW  CHECKLIST) 


ENVIRONMENTAL  IMPACT 

Has  an  environmental  impact  study  been  completed  (to  determine  if 
the  system  will  have  an  adverse  impact  on  the  environment)? 

This  shall  be  coordinated  with  the  training  center/post 
Directorate  of  Engineering  and  Housing  (DEH)  NEPA  Coordinator;  a 
record  of  environmental  consideration  shall  be  completed  in 
accordance  with  AR  200-2  and  the  National  Environmental 
Protection  Act  (NEPA) . 

Are  the  required  standards  associated  with  air  quality,  water 
quality,  and  noise  levels  being  maintained  in  spite  of  the  intro¬ 
duction  and  operation  of  the  TES  System? 

Have  potentially  degrading  ecological  effects  been  identified? 

Has  corrective  action  been  taken  to  eliminate  potential  problems? 
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APPENDIX  D 


SAMPLE  DATABASE  TRACKING  SHEETS  AND  OUTPUTS 
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S_  Target 

;;  Type  ' 

Warhead 

Type 

;  V^£r(8) 
••  -Firecr k 

-• -  ■ST’pUrfi  -**T 1 

-No-  of  ^ 

^Engagements  J^Hits.>- 

Demonstrated  ! 

Soft  Skin 

PD 

1 

0 

0 

0.0000 

Soft  Skin 

PD 

2 

0 

0 

0.0000 

Soft  Skin 

PD 

3 

11 

1 

0.0909 

Soft  Skin 

PD 

4 

7 

1 

0.1429 

Soft  Skin 

MP 

1 

0 

0 

0.0000 

Soft  Skin 

MP 

2 

0 

0 

0.0000 

Soft  Skin 

MP 

3 

9 

0 

0.0000 

Soft  Skin 

MP 

4 

0 

0 

0.0000 

Constraints 

Constraints 

Met=l 

not  Met=0 

Hit=l  (at  least  one  pair  hit  during  engagement) 
Miss=0  (no  pair(s)  hit  during  engagement) 

Date 

Acft 

Constraints 

”  Target 

Warhead 

Pair (s)  ! 

Hit 

Ser  No . 

}y\ -.1 

'  -Type 

Type  - 

riiivil  Fired 

04-Mar-98 

96-5001 

l 

Soft 

PD 

4 

0 

04-Mar-98 

96-5001 

l 

Soft 

MP 

3 

0 

04-Mar-98 

96-5001 

l 

Soft 

PD 

3 

0 

04-Mar-98 

96-5008 

l 

Soft 

PD 

4 

0 

04-Mar-98 

96-5008 

l 

Soft 

MP 

3 

0 

04-Mar-98 

96-5008 

l 

Soft 

PD 

3 

1 

04-Mar-98 

97-5027 

l 

Soft 

PD 

4 

1 

04-Mar-98 

97-5027 

l 

Soft 

MP 

3 

0 

04-Mar-98 

97-5027 

l 

Soft 

PD 

3 

0 

04-Mar-98 

96-5014 

0 

Soft 

PD 

4 

0 

04-Mar-98 

96-5014 

l 

Soft 

MP 

3 

0 

04-Mar-98 

96-5014 

l 

Soft 

PD 

3 

0 

04-Mar-98 

96-5010 

l 

Soft 

PD 

4 

0 

04-Mar-98 

96-5010 

l 

Soft 

MP 

3 

0 

04-Mar-98 

96-5010 

l 

Soft 

PD 

3 

0 

04-Mar-98 

97-5025 

l 

Soft 

PD 

3 

0 

04-Mar-98 

97-5025 

l 

Soft 

MP 

3 

0 

04-Mar-98 

97-5025 

l 

Soft 

PD 

3 

0 

04-Mar-98 

96-5009 

l 

Soft 

PD 

4 

0 

04-Mar-98 

96-5009 

0 

-  - •  Soft 

MP 

3 

0 

04-Mar-98 

96-5009 

1 

Soft 

PD 

3 

0 

07-Mar-98 

96-5002 

1 

Soft 

PD 

4 

0 

07-Mar-98 

96-5002 

1 

Soft 

MP 

3 

0 

07-Mar-98 

96-5002 

1 

Soft 

PD 

3 

0 

07-Mar-98 

96-5009 

1 

Soft 

PD 

4 

0 

07-Mar-98 

96-5009 

0 

Soft 

MP 

3 

0 

07-Mar-98 

96-5009 

1 

Soft 

PD 

3 

0 

Figure  D-l 

Example  of  Rocket  Event  Tracking  Sheet 
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Example  of  Database  Output  for  Pilots'  Questionnaire 
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Example  of  Database  Output  for  Criterion  Scores 
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Example  MANPRINT  Database  Tracking  Sheet  for  Installation/De-installation 


14:55 


Example  RAM  Database  Tracking  Sheet  (for  Individual  TES  System) 


VITA 
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